search

3liz / lizmap-wps-web-client-module

Lizmap module for WPS Client support
Mozilla Public License 2.0
11 stars 9 forks source link

Lizmap web client with py-qgis-server and py-qgis-wps #21

Closed prahout59 closed 2 years ago

prahout59 commented 2 years ago

First, many thanks for your job, I really like to work with Lizmap.

We are using lizmap web client v3.3 with qgis server and ws-cgi in our production server. I want to create graphical modeles with Qgis Desktop and export them in a wps server that can treat the process after getting parameters to restitute a layer.

First, on my testing server, I’ve replaced qgis-fcgi with your Qgis server: https://github.com/3liz/py-qgis-server

My configuration file for this service is the following /usr/local/etc/qgisserver.ini :

[server]
port = 8080
timeout = 60
[logging]
level = DEBUG
[projects.cache]
rootdir = /home/qgis/projects/
[api.enabled]
landing_page = yes

I’ve disabled the fcgi configuration in apache and modified lizmapConfig.ini.php to use py-qgis-server rather than fcgi mode. lizmapConfig.ini.php modification

; wmsServerURL="http://127.0.0.1/cgi-bin/qgis_mapserv.fcgi"
wmsServerURL="http://127.0.0.1:8080/ows"

All is ok and I can see my projects like usual.

Second, I’ve installed your wps server : https://github.com/3liz/py-qgis-wps

My configuration file for this service is /usr/local/etc/wpsserver.ini : 

[server]
port = 8081
[projects.cache]
rootdir = /home/qgis/projects
[logging]
level = DEBUG

I’ve made a test of the service with the url : http://localhost:8081/ows/status/?SERVICE=WPS and the response is {"status": []} which seems Ok.

The next step is to configure Lizmap web client according to those two new services and put the graphical modeler scripts in the good place.

I’ve modified the localConfig.ini.php with wps.access=2

[modules]
;; uncomment it if you want to use ldap for authentication
;; see documentation to complete the ldap configuration
jacl2.access=1
jauth.access=2
jauthdb.access=1
ldapdao.access=1
cadastre.access=2
wps.access=2

and a new section called [wps]

[wps]
wps_url=http:8081/ows
wps_rootDirectories=/projects/wps
redis_host=localhost
redis_port=6379
redis_key_prefix=lzmwps
ows_url=http:8080/ows

Please, could you confirm I’m Ok and tell me where to put my processes files, in which format (python or native qgis or in project file).

I think I’m not so far from the solution, but I need your help to put the files in the good places.

prahout59 commented 2 years ago

Je me suis rendu compte de l'erreur dans mon localconfig.ini.php. J'ai modifié les urls comme suit :

[wps]
wps_url=http://localhost:8081
wps_rootDirectories=/projects/wps
redis_host=localhost
redis_port=6379
redis_key_prefix=lzmwps
ows_url=http://localhost:8080/ows

Je vais tester mais je ne sais toujours pas où stocker les fichiers issus do modeleur graphique.

dmarteau commented 2 years ago

Hi

Once you have configured the provider's module path, simply creates a models directory at this location and drop your model's file in it.

Note that is the same for processing scripts.

Your processing module directory should be something like:

<PROCESSING_PROVIDERS_MODULE_PATH>/
     \ models/
           \ <your models files here>
     \ scripts/
           \ <your scripts files here>

The rest of your lizmap configuration seems ok. Note that you may find examples here and here (with a complete docker compose stack)

dmarteau commented 2 years ago

@prahout59 Take care to bind wps results data to py-qgi-server since lizmap-web client use py-qgis-server to serve wps results as WMS/WFS services.

In the py-qgis-wps configuration, defines where to store results

[server]
workdir = <absolute_path_to_stored_results_data>

In the py-qgis-server configuration:

[projects.schemes]
wps_results = <absolute_path_to_stored_results_data>

See documentation on custom schemes alias

prahout59 commented 2 years ago

I think I've still a problem with the wps server. When I make a test with http://localhost:8081/?service=wps&version=1.0.0&request=GetCapabilities, the response is 

{"author": "3Liz", "author_url": "http://3liz.com", "config": {"DEFAULT": {}, "logging": {"level": "DEBUG"}, "logstorage:redis": {"dbnum": "0", "host": "localhost", "port": "6379", "prefix": "pyqgiswps"}, "metadata:main": {"contact_address": "Mailing Address", "contact_email": "Email Address", "contact_name": "Lastname, Firstname", "contact_role": "pointOfContact", "contact_url": "Contact URL", "identification_abstract": "Py-Qgis-WPS is an implementation of the Web Processing Service standard from the Open Geospatial Consortium. Py-Qgis-WPS is written in Python.", "identification_accessconstraints": "NONE", "identification_fees": "NONE", "identification_keywords": "Py-Qgis-WPS,WPS,OGC,QGIS,processing", "identification_keywords_type": "theme", "identification_title": "Py-Qgis-WPS Processing Service", "provider_name": "Organization Name", "provider_url": "https://github.com/3liz/py-qgis-wps"}, "processing": {"accesspolicy": "/home/qgis/wps/accesspolicy.yml", "adjust_ellipsoid": "no", "default_crs": "EPSG:4326", "default_module_path": "/usr/share/qgis_wps/processing", "destination_root_path": "./", "exposed_providers": "script,model", "providers_module_path": "/home/qgis/wps", "raster.fileext": "", "unsafe.raw_destination_input_sink": "no", "vector.fileext": ""}, "projects.cache": {"rootdir": "/home/qgis/projects", "size": "10", "strict_check": "yes"}, "projects.schemes": {}, "qgis.projects": {"wmsurl": "${server:wms_service_url}"}, "qgis.settings": {}, "qgis.settings.folders": {"Processing/Configuration/MODELS_FOLDER": "/home/qgis/wps/models", "Processing/Configuration/SCRIPTS_FOLDERS": "/home/qgis/wps/scripts"}, "server": {"cleanup_interval": "600", "cross_origin": "yes", "download_ttl": "30", "enable_filters": "yes", "encoding": "utf-8", "host_proxy": "", "interfaces": "0.0.0.0", "language": "en-US", "maxbuffersize": "1m", "maxinputsize": "100m", "maxqueuesize": "100", "outputfile_as_reference": "yes", "parallelprocesses": "1", "port": "8081", "processlifecycle": "1", "response_expiration": "86400", "response_timeout": "1800", "restartmon": "", "ssl": "no", "ssl_cert": "", "ssl_key": "", "status_url": "{host_url}ows/?service=WPS&request=GetResults&uuid={uuid}", "store_url": "{host_url}store/{uuid}/{file}?service=WPS", "sys_config_path": "/usr/share/qgis_wps", "url": "{host_url}", "wms_response_url": "${wms_service_url}?MAP={map_url}&service=WMS&request=GetCapabilities", "wms_service_url": "${wps.request:host_url}", "workdir": "/home/qgis/results", "wps_result_map_uri": "wps-results:"}, "wps.request": {}}, "qgis_version": "31803 (Z\u00fcrich)", "styles": {}, "tornado_ver": "6.1", "version": "n/a"}

I've no service listed in the response even if I've transfered a model file in my wps structure which is /home/qgis/wps/models

prahout59 commented 2 years ago

Now, wps server can see my .model3 file but

2021-09-22 17:36:15,255 [4761]  DEBUG   Loading processing algorithms from provider 'model'2021-09-22 17:36:15,256 [4761]  INFO    model:patrick: dropping hidden param: VERBOSE_LOG
But my model which name is patrick.model3 seems to be dropped and so I can't view it. I have not seen a model's parameter in Qgis desktop that allow me to make it hidden or not.

2021-09-22 17:42:54,126 [4920]  DEBUG   Loading processing algorithms from provider 'model'
2021-09-22 17:42:54,126 [4920]  INFO    model:patrick: dropping hidden param: VERBOSE_LOG
prahout59 commented 2 years ago

Here after the complete trace for my wps instance. There is some warning messages about No WPS plugin found ?

root@debianpra:~# root@debianpra:~# wpsserver -c /usr/local/etc/wpsserver.ini
Failed to read manifest !: [Errno 2] No such file or directory: '/root/py-qgis-wps/pyqgiswps/build.manifest'
wpsserver n/a (build n/a,commit n/a)
loading configuration
Configuration file </usr/local/etc/wpsserver.ini> loaded
Log level set to DEBUG

2021-09-22 17:42:53,372 [4916]  INFO    Running WPS server on port 0.0.0.0:8081
2021-09-22 17:42:53,372 [4916]  DEBUG   Looking for plugins in /usr/share/qgis_wps/processing
2021-09-22 17:42:53,373 [4916]  WARNING No WPS plugin found in /usr/share/qgis_wps/processing
2021-09-22 17:42:53,373 [4916]  DEBUG   Looking for plugins in /home/qgis/wps
2021-09-22 17:42:53,373 [4916]  WARNING No WPS plugin found in /home/qgis/wps
2021-09-22 17:42:53,379 [4920]  DEBUG   *** Folder settings: Processing/Configuration/SCRIPTS_FOLDERS = /home/qgis/wps/scripts
2021-09-22 17:42:53,380 [4920]  INFO    Processing/Configuration/SCRIPTS_FOLDERS = /home/qgis/wps/scripts
2021-09-22 17:42:53,380 [4920]  DEBUG   *** Folder settings: Processing/Configuration/MODELS_FOLDER = /home/qgis/wps/models
2021-09-22 17:42:53,380 [4920]  INFO    Processing/Configuration/MODELS_FOLDER = /home/qgis/wps/models
2021-09-22 17:42:53,659 [4920]  INFO    Starting Qgis application: 3.18.3-Zürich
2021-09-22 17:42:53,660 [4920]  INFO    Setting offscreen mode
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
2021-09-22 17:42:53,891 [4920]  INFO    [qgis:4920] Installing cleanup hook
Application state:
QGIS_PREFIX_PATH env var:
Prefix:         /usr/
Plugin Path:            /usr//lib/qgis/plugins
Package Data Path:      /usr//share/qgis
Active Theme Name:
Active Theme Path:      /usr//share/qgis/resources/themes//icons/
Default Theme Path:     :/images/themes/default/
SVG Search Paths:       /usr//share/qgis/svg/
                /root/.local/share/QGIS/QGIS3/profiles/default/svg/
User DB Path:   /usr//share/qgis/resources/qgis.db
Auth DB Path:   /root/.local/share/QGIS/QGIS3/profiles/default/qgis-auth.db

2021-09-22 17:42:53,891 [4920]  INFO    [qgis:4920] Qgis application initialized......
/usr/lib/python3/dist-packages/osgeo/__init__.py:8: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
2021-09-22 17:42:54,082 [4920]  ERROR   [qgis:4920] Processing: Problem with GRASS installation: GRASS was not found or is not correctly installed
2021-09-22 17:42:54,097 [4920]  ERROR   [qgis:4920] Processing: Problem with SAGA installation: SAGA was not found or is not correctly installed
2021-09-22 17:42:54,110 [4920]  INFO    [qgis:4920] QGis processing initialized
2021-09-22 17:42:54,111 [4920]  DEBUG   = Registering provider 'script'
2021-09-22 17:42:54,111 [4920]  DEBUG   = Registering provider 'model'
2021-09-22 17:42:54,119 [4920]  DEBUG   Registering factory: @3liz.org/cache/protocol-handler;1?scheme=file
2021-09-22 17:42:54,119 [4920]  DEBUG   Registering factory: @3liz.org/cache/protocol-handler;1?scheme=postgres
2021-09-22 17:42:54,120 [4920]  DEBUG   Registering factory: @3liz.org/cache-manager;1
2021-09-22 17:42:54,126 [4920]  DEBUG   Loading processing algorithms from provider 'script'
2021-09-22 17:42:54,126 [4920]  DEBUG   Loading processing algorithms from provider 'model'
2021-09-22 17:42:54,126 [4920]  INFO    model:patrick: dropping hidden param: VERBOSE_LOG
2021-09-22 17:42:54,127 [4920]  DEBUG   validator: <function emptyvalidator at 0x7ff62fe0ad90>
2021-09-22 17:42:54,127 [4920]  DEBUG   validator: <function emptyvalidator at 0x7ff62fe0ad90>
2021-09-22 17:42:54,127 [4920]  DEBUG   validator: <function emptyvalidator at 0x7ff62fe0ad90>
2021-09-22 17:42:54,127 [4920]  DEBUG   validator: <function emptyvalidator at 0x7ff62fe0ad90>
2021-09-22 17:42:54,127 [4920]  DEBUG   validator: <function emptyvalidator at 0x7ff62fe0ad90>
2021-09-22 17:42:54,127 [4920]  DEBUG   validator: <function emptyvalidator at 0x7ff62fe0ad90>
2021-09-22 17:42:54,127 [4920]  INFO    Published processes:
 * model:patrick
2021-09-22 17:42:54,417 [4916]  DEBUG   Registering factory: @3liz.org/cache/protocol-handler;1?scheme=file
2021-09-22 17:42:54,422 [4916]  DEBUG   Registering factory: @3liz.org/cache/protocol-handler;1?scheme=postgres
2021-09-22 17:42:54,422 [4916]  DEBUG   Registering factory: @3liz.org/cache-manager;1
Application path not initialized
2021-09-22 17:42:54,574 [4916]  DEBUG   LOGSTORE: Initializing REDIS session
2021-09-22 17:42:54,578 [4928]  INFO    Starting worker pool
2021-09-22 17:42:54,579 [4916]  DEBUG   Started server
2021-09-22 17:42:54,585 [4931]  DEBUG   Connecting to ipc:///tmp/qgswps/pooladdr_4916
2021-09-22 17:42:54,588 [4931]  DEBUG   Identity set to b"\xc0\x9b]h\x1b\xbb\x11\xec\xa2\x9d\x08\x00'5c+"
2021-09-22 17:42:54,589 [4931]  DEBUG   Enabling broadcast notification
2021-09-22 17:42:54,589 [4916]  INFO    Starting supervisor
2021-09-22 17:42:54,590 [4916]  INFO    WPS Server ready
2021-09-22 17:42:54,591 [4931]  DEBUG   *** Folder settings: Processing/Configuration/SCRIPTS_FOLDERS = /home/qgis/wps/scripts
2021-09-22 17:42:54,591 [4931]  INFO    Processing/Configuration/SCRIPTS_FOLDERS = /home/qgis/wps/scripts
2021-09-22 17:42:54,592 [4931]  DEBUG   *** Folder settings: Processing/Configuration/MODELS_FOLDER = /home/qgis/wps/models
2021-09-22 17:42:54,594 [4931]  INFO    Processing/Configuration/MODELS_FOLDER = /home/qgis/wps/models
2021-09-22 17:42:54,595 [4931]  INFO    Starting Qgis application: 3.18.3-Zürich
2021-09-22 17:42:54,596 [4931]  INFO    Setting offscreen mode
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
2021-09-22 17:42:54,776 [4931]  INFO    [qgis:4931] Installing cleanup hook
Application state:
QGIS_PREFIX_PATH env var:
Prefix:         /usr/
Plugin Path:            /usr//lib/qgis/plugins
Package Data Path:      /usr//share/qgis
Active Theme Name:
Active Theme Path:      /usr//share/qgis/resources/themes//icons/
Default Theme Path:     :/images/themes/default/
SVG Search Paths:       /usr//share/qgis/svg/
                /root/.local/share/QGIS/QGIS3/profiles/default/svg/
User DB Path:   /usr//share/qgis/resources/qgis.db
Auth DB Path:   /root/.local/share/QGIS/QGIS3/profiles/default/qgis-auth.db

2021-09-22 17:42:54,777 [4931]  INFO    [qgis:4931] Qgis application initialized......
2021-09-22 17:42:54,814 [4931]  ERROR   [qgis:4931] Processing: Problem with GRASS installation: GRASS was not found or is not correctly installed
2021-09-22 17:42:54,832 [4931]  ERROR   [qgis:4931] Processing: Problem with SAGA installation: SAGA was not found or is not correctly installed
2021-09-22 17:42:54,844 [4931]  INFO    [qgis:4931] QGis processing initialized
2021-09-22 17:42:54,845 [4931]  DEBUG   = Registering provider 'script'
2021-09-22 17:42:54,845 [4931]  DEBUG   = Registering provider 'model'
2021-09-22 17:42:54,845 [4931]  DEBUG   Starting ZMQ worker loop
2021-09-22 17:42:54,846 [4916]  DEBUG   WORKER READY b"\xc0\x9b]h\x1b\xbb\x11\xec\xa2\x9d\x08\x00'5c+"
2021-09-22 17:43:42,155 [4916]  REQ     10.0.2.2        200     GET     /?service=wps&version=1.0.0&request=GetCapabilities&map=patrick        10       2643    Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36
dmarteau commented 2 years ago

Published processes: model:patrick

Your model is correctly registered

When I make a test with http://localhost:8081/?service=wps&version=1.0.0&request=GetCapabilities, the response is...

This url is wrong, what your get is the root / response: the service run at http://localhost:8081/ows/

prahout59 commented 2 years ago

It's Ok with http://localhost:8081/ows/?service=wps&version=1.0.0&request=GetCapabilities

I think, now, I just have to create a policy.yml in /home/qgis/wps and lizmap will see the models

prahout59 commented 2 years ago

I've created the file /home/qgis/wps/policy.yml whith allow for everybody and now, I can see my model and my script. 

autoreload: yes

# Define access policies
policies:
    # Global policy
    - allow: all

# Include other policies
#include_policies:
#    - policy/*/*.yml
Gustry commented 2 years ago

@prahout59 You can make nice syntax coloring in your post when you copy/paste config : ini, yml, bash ... It will avoid markdown rendering #.

prahout59 commented 2 years ago

@Gustry Ok, I'll take care about my syntax coloring, you're right

dmarteau commented 2 years ago

I've created the file /home/qgis/wps/policy.yml whith allow for everybody and now, I can see my model and my script.

You don't really need a policy if you give acces to everybody.

prahout59 commented 2 years ago

Without the policy file, no models appear in lizmap Web client. I'll modifiy this file to give the good rights later.

@dmarteau Many thanks for your help, indeed.