processing.runAndLoadResults does not load results

m-rm commented 1 year ago

I have recently started using QGIS WPS. I am using the docker image 3liz/qgis-wps:3.30. For testing I wrote a very simple algorithm with no INPUT or OUTPUT:

        feedback.pushDebugInfo("Step 1")
        res1 = processing.runAndLoadResults(
            "qgis:rastercalculator",
            {
                'EXPRESSION': '1',
                'LAYERS': None,
                'CELLSIZE': 10,
                'EXTENT': '67740.870400000,79959.890400000,341924.860400000,360502.590400000 [EPSG:31255]',
                'CRS': QgsCoordinateReferenceSystem('EPSG:31255'),
                'OUTPUT': '/tmp/test1.tiff'}, context=context, feedback=feedback)

        feedback.pushDebugInfo("Step 2")
        res2 = processing.run(
            "qgis:rastercalculator",
            {
                'EXPRESSION': '"test1@1"',
                'LAYERS': None,
                'CELLSIZE': 10,
                'EXTENT': '67740.870400000,79959.890400000,341924.860400000,360502.590400000 [EPSG:31255]',
                'CRS': QgsCoordinateReferenceSystem('EPSG:31255'),
                'OUTPUT': 'TEMPORARY_OUTPUT'}, context=context, feedback=feedback)

it simply creates a raster layer and uses it in the next step to create a second raster layer. It works absolutely fine when I run it in QGIS Desktop, but when I run it in through WPS I get the following log:

2023-04-25 07:11:57,357 [28]    INFO    Starting task 770e7536:script:test
2023-04-25 07:11:57,360 [28]    DEBUG   Resolving 'file' protocol
2023-04-25 07:11:57,362 [28]    DEBUG   Reading Qgis project /projects/klimalinz/KLIMALINZ_v1.4.qgz
2023-04-25 07:11:59,050 [28]    DEBUG   script:test:770e7536 Step 1
2023-04-25 07:11:59,082 [28]    INFO    script:test:770e7536 Results: {'OUTPUT': '/tmp/test1.tiff'}
2023-04-25 07:11:59,146 [28]    DEBUG   script:test:770e7536 Step 2
2023-04-25 07:11:59,184 [28]    CRITICAL    Traceback (most recent call last):
  File "/opt/local/pyqgiswps/lib/python3.9/site-packages/pyqgiswps/executors/processingprocess.py", line 242, in run_processing_algorithm
    results, ok = alg.run(parameters, context, feedback, configuration=create_context,
_core.QgsProcessingException: An error occurred while performing the calculation

2023-04-25 07:11:59,184 [28]    INFO    script:test:770e7536 memory: start=183.469Mb end=213.340Mb delta=29.871Mb

I investigated a little further and the problem is that runAndLoadResults does not load the output layer. When I instead use the following code for step 1 it works fine:

        res1 = processing.run(
            "qgis:rastercalculator",
            {
                'EXPRESSION': '1',
                'LAYERS': None,
                'CELLSIZE': 10,
                'EXTENT': '67740.870400000,79959.890400000,341924.860400000,360502.590400000 [EPSG:31255]',
                'CRS': QgsCoordinateReferenceSystem('EPSG:31255'),
                'OUTPUT': 'TEMPORARY_OUTPUT'}, context=context, feedback=feedback)

        outputlayer = QgsRasterLayer(res1['OUTPUT'], 'test1')
        QgsProject.instance().addMapLayer(outputlayer)

Since it works in QGIS Desktop but not in WPS I assume this might be a bug.

I use the following docker-compose.yml:

version: '3.8'
services:
  wps:
    image: 3liz/qgis-wps:3.30
    environment:
      QGSWPS_SERVER_PARALLELPROCESSES: '2'
      QGSWPS_SERVER_LOGSTORAGE: REDIS
      QGSWPS_REDIS_HOST: redis
      QGSWPS_PROCESSING_PROVIDERS_MODULE_PATH: /processing
      QGSWPS_CACHE_ROOTDIR: /projects
      QGSWPS_SERVER_WORKDIR: /srv/data
      QGSWPS_USER: 1000:1000
      QGSWPS_LOGLEVEL: DEBUG
    volumes:
      - ./processing:/processing
      - ./projects:/projects
      - ./data:/srv/data
      - ./qgis:/home/qgis
    ports:
      - 127.0.0.1:8888:8080
  redis:
      image: redis:5-alpine

Complete algorithm: https://pastebin.com/8EcRwdVi

dmarteau commented 1 year ago

Contexts are somewhat different about how results are handled in Qgis Desktop and py-qgis-wps:

In Qgis desktop, output results are automatically added in to the current Project context and I guess that that the expression used in your second calculation in your algorithm refers to to the output layer of calculation 1.

In py-qgis-wps, the source project is never modified implicitely for obvious reason that a given process should be idempotent. Output layers must be handled explicitely (Output layer parameters are added to a destination project different from the source project).

So I suggest that that you explicitely pass the output of the first algorithm as input of the second one, instead of assuming that the first output is appended implicitely to the source project - which is never the case in py-qgis-wps.

m-rm commented 1 year ago

Thank you for your answer. Unfortunately qgis:rastercalculator does not have an INPUT parameter, just a LAYERS parameter, which is used as input, but additionally is used to calculate CELLSIZE, EXTENT and CRS if they are not specified explicitly. I have some cases where I want to have this calculated and therefore don't want to use LAYERS for input, since it would change the calculation.

So I have 2 options:

Either add the output layers to the source project with QgsProject.instance().addMapLayer(outputlayer)
Copy all input layers from the source project to the output project and don't use context as parameter for processing.run(...)

m-rm commented 1 year ago

Output layer parameters are added to a destination project different from the source project

The destination project is the one that is available in the Resources tab after successful execution? I have the problem that this project is always empty without any layers. I'm using the following code:

    def processAlgorithm(self, parameters, context, feedback):
        res = processing.runAndLoadResults(
            "qgis:rastercalculator",
            {
                'EXPRESSION': '1',
                'LAYERS': None,
                'CELLSIZE': 10,
                'EXTENT': '67740.870400000,79959.890400000,341924.860400000,360502.590400000 [EPSG:31255]',
                'CRS': QgsCoordinateReferenceSystem('EPSG:31255'),
                'OUTPUT': "test.tiff"}, context=context, feedback=feedback)

        return {'OUTPUT': res['OUTPUT']}

The created layer should have been added to the destination project, right? The log shows this:

2023-04-26 09:16:18,198 [28]    INFO    Starting task 007a3320:script:test
2023-04-26 09:16:18,202 [28]    DEBUG   Resolving 'file' protocol
2023-04-26 09:16:18,204 [28]    DEBUG   Reading Qgis project /projects/klimalinz/KLIMALINZ_v1.4.qgz
2023-04-26 09:16:19,062 [28]    INFO    script:test:007a3320 Results: {'OUTPUT': 'test.tiff'}
2023-04-26 09:16:19,082 [28]    INFO    script:test:007a3320 Results: {'OUTPUT': 'test.tiff'}
2023-04-26 09:16:19,095 [28]    DEBUG   Layer name set to Output <details name was: Output>
2023-04-26 09:16:19,096 [28]    DEBUG   Getting style for script:test: OUTPUT <None>
2023-04-26 09:16:19,097 [28]    DEBUG   Adding Map layer 'test.tiff' (outputName OUTPUT) to Qgs Project
2023-04-26 09:16:19,102 [28]    DEBUG   Writing Results to /srv/data/007a3320-e413-11ed-819a-aed3312cc93a
2023-04-26 09:16:19,112 [28]    INFO    Task finished script:test:007a3320-e413-11ed-819a-aed3312cc93a
2023-04-26 09:16:19,173 [28]    INFO    script:test:007a3320 memory: start=183.305Mb end=211.309Mb delta=28.004Mb

where Adding Map layer 'test.tiff' (outputName OUTPUT) to Qgs Project also indicates that it has been added. The following files were created:

grafik

but when I open script_test.qgs in QGIS Desktop it doesn't have any layers. When I open the file in an editor it doesn't contain any reference to test.tiff.

Am I doing something wrong?

dmarteau commented 1 year ago

but when I open script_test.qgs in QGIS Desktop it doesn't have any layers. When I open the file in an editor it doesn't contain any reference to test.tiff.

I guess because the destination project is not correct: you may check the destination project with the following snippet

for layername, details in context.layersToLoadOnCompletion().items():
    // You may log out `details.project.fileName()` and `context.destination_project.fileName()`

context.destination_project must be the script_test.qgs project
details.project is the project destination where the layer will be added,

if details.project is null, py-qgis-server will use the context.destination_project. Usually details.project is already set to the destination project. If they are not the same this may be an indication the algorithm has overriden details.project somehow.

m-rm commented 1 year ago

details.project.fileName() is None and context.destination_project.fileName() is None. What could be the cause?

EDIT: If I manually add the layer to context.destination_project it works.

dmarteau commented 1 year ago

details.project.fileName() is None and context.destination_project.fileName() is None. What could be the cause?

My bad: this is because the project has still not beeing assigned to a file, sorry. So it is not suprising that fileName() is None. So the best is to check that details.project and context.destination_project is the same object.

EDIT: If I manually add the layer to context.destination_project it works.

This is what is supposed to happen when it says: Adding Map layer 'test.tiff' (outputName OUTPUT) to Qgs Project. Does the destination project reference any layer ?

m-rm commented 1 year ago

I checked if the projects match with this code:

        for layername, details in context.layersToLoadOnCompletion().items():
            feedback.pushDebugInfo(f"{layername} project: {details.project is context.destination_project}")

the output is 2023-04-27 12:31:53,577 [31324] DEBUG script:test:b9c28b4c test.tiff project: False so no, the projects do not match.

EDIT: I just tried this:

        for layername, details in context.layersToLoadOnCompletion().items():
            feedback.pushDebugInfo(f"{layername} project: {details.project is QgsProject.instance()}")

and it printed True, so the layer is assigned to the input project instead of the destination project.

3liz / py-qgis-wps

processing.runAndLoadResults does not load results #40