Upgrade to GeoNode with QGIS Server backend

cgiovando commented 7 years ago

As already discussed, GeoNode with QGIS Server backend developed by Kartoza (https://github.com/kartoza/geonode) should be installed for the Lab's instance and support any data request from ThinkHazard.

Also relevant to this:

Please coordinate with @gubuntu and Kartoza team for any question or issues. Thanks!

cgiovando commented 7 years ago

@fvanderbiest how is this going? Were you successful in deploying GeoNode with QGIS server?

fvanderbiest commented 7 years ago

Were you successful in deploying GeoNode with QGIS server?

Still in the works... We'll let you know when we encounter blocking issues.

how is this going?

We were wondering how we could migrate the existing geoserver "datadir" into qgis server. Any hints @gubuntu ? Thanks.

gubuntu commented 7 years ago

I depends what is there. Although QGIS can handle any data source, as a GeoNode/GeoSAFE backend it is limited to shape files, GeoTIFF and raster asc files.

I would script the manage.py to walk through the data dir and load each of those layers into your instance of geonode-qgis_server. This will result in them being properly processed and published. Some things that happen during that process:

a QGIS project is created per layer with the otf server plugin
QGIS generates a TMS cache for each layer
if a .qml file accompanies the file then that style will be used, otherwise QGIS will create a default style.
if you want to use the data in GeoSAFE (which we do in this case, for all valid hazard and exposure layers) then each layer will need properly structured InaSAFE 3.5 keywords in the Supplemental Information element of an ISO 19115 metadata xml file.

So, some preparation before loading would be necessary:

generate a qml for each layer from the existing sld. You could automate this through the QGIS API. This would work for vectors. QGIS can't apply SLD to rasters so you might need to style the layer in QGIS desktop and export the qml.
generate InaSAFE 3.5 keywords in Desktop QGIS for each layer and export the metadata xml file.

The qml and metadata files must have the same name as the data file and sit next to it on disk.

do you agree with or can you expand on my steps @gustry @timlinux @lucernae?

fvanderbiest commented 7 years ago

Were you successful in deploying GeoNode with QGIS server?

Unfortunately not. We failed to make it work in a way which is compatible with our container orchestration (running rancher with cattle).

Here are some hints to get a production-ready composition :

use static images (no code mounted locally), regularly built by docker hub or any other (travic CI does it too), with proper tagging (no "latest" tag)
use named volumes for persistent data or anonymous volumes for temporary data, no local mount points, no volumes_from
do not rely on IPs but container names
logs should be redirected to stdout/stderr, not to log files (or in a named volume, not in the container - container writes are known to be slow)

I'm probably missing several others too...

These hints also apply to the base geonode composition. We're working on it, eg with https://github.com/GeoNode/geonode/pull/3076

lucernae commented 7 years ago

Hi @fvanderbiest I'm aware that this orchestration were not rancher ready, but didn't know that you guys were aiming for this.

I've been using rancher too a couple of weeks ago to try deploy this using rancher-cattle and I think I at least understand about these hints and it was on point. If what we aim is having a rancher-ready container, then what I can propose to @gubuntu is for us to create a separate "production" mode orchestration for us to build and push the image to docker hub, at least for the different service first.

We rely heavily on host mounting in github.com/kartoza/docker-geosafe for development reason, of course. I still need to learn how to correctly store data in named volumes and how to maintain it. For example, do we need to store postgis directory to named volume? Or how do we maintain Geonode's uploaded directory. Do you have any production orchestration (rancher-ready) that I can learn from? It will be good for me to learn this too.

These hints also apply to the base geonode composition. We're working on it, eg with GeoNode#3076

Thank's for working on this.

fvanderbiest commented 7 years ago

but didn't know that you guys were aiming for this.

Yes, the border between dev and ops is thin and continuously moving ... It's difficult for us to build production-ready images when everything's moving and we're not in charge of the code.

to create a separate "production" mode orchestration for us to build and push the image to docker hub

This is the way to go, yes.

I still need to learn how to correctly store data in named volumes and how to maintain it.

That's easy. IMO, it is important to have in mind that:

named_volume:/path/to/image/folder => named_volume gets bootstrapped with /path/to/image/folder contents.
data is safe in named volumes, even when the container is deleted. It does not get deleted. It stays on the hosts as long as no one invokes docker volume rm xx
named volumes backups can be accomplished through the use of eg https://github.com/camptocamp/conplicity

For example, do we need to store postgis directory to named volume

That's correct. See eg https://github.com/georchestra/docker/blob/master/docker-compose.yml#L15-L21

Do you have any production orchestration (rancher-ready) that I can learn from?

Yes, we're actively working on another SDI with eg https://github.com/georchestra/docker. This repo contains the production-ready composition. Unfortunately, the rancher stacks leveraging this are closed-source for now.

geOrchestra images are built from the repo hosting the source code and there's a makefile: make docker-build

fvanderbiest commented 7 years ago

Yes, we're actively working on another SDI with eg https://github.com/georchestra/docker.

Well, the example is not that good, since we're mounting the local config folder into the images. Instead it should be a named volume.

fvanderbiest commented 7 years ago

These hints also apply to the base geonode composition. We're working on it, eg with GeoNode#3076

Also improved with https://github.com/camptocamp/labs_geonode/pull/4 (we're planning to contribute it soon upstream)

vdeparday commented 6 years ago

@cgiovando @gubuntu @fvanderbiest is the QGIS-Server backend now fully rancher ready?

Before moving forward with this, I would like to have a clear documentation and diagnostic of the implications (permissions, client, link with ThinkHazard! harvesting process, etc...) of the move to QGIS-Server and a more general summary of the pros, cons, potential issues linking with ThinkHazard.

@cgiovando can you take the lead on that taking the input from everyone?

stufraser1 commented 5 years ago

@cgiovando where are we at on this? should this remain open?

cgiovando commented 5 years ago

I think we can close it for now, until we decide data hosting strategy across the different Lab's products.

GFDRR / geonode

Upgrade to GeoNode with QGIS Server backend #37