UR 13: Allow cloud processing for the generation of scenarios

[danangmassandy]

Expectation:

• As a user, I want the ability to use cloud processing to generate scenarios in a practical time frame, so that I can make decisions more quickly and show the plugin in a workshop setting.

Acceptance Criteria:

• The plugin includes a feature to use cloud processing.
• An API is created to do the cloud processing.
• The cloud processing happens quickly and efficiently.

Tasks:

• [x] [8] Integrate PyQGIS to the worker
• [x] [20] Refactor cplus plugin code to be able to run using pyqgis
• [x] [8] Configure input parameters for the cplus plugin code
• [x] [5] Load layer inputs from storage (S3)
• [x] [5] Upload layer scenario outputs to the storage (S3)
• [x] [5] Update progress and log of the processing (related to UR 19)
• [x] [2] Compare the performance of the analysis processing between worker and QGIS desktop in different scenarios
• [ ] TODO: change the qgis processing to alternative tools for performance comparison (see https://github.com/kartoza/cplus-api/issues/2)
• [x] [3] Task error handling, when task is stopped with error, the system should be able to clean the temporary files for this task.

The task to run the analysis should have below considerations:

• Memory efficient, please ensure that there is no memory leak after the task is finished.
• Disk resource efficient usage, the worker container will have limited internal storage, so we should manage the input layer that needs to be downloaded right before it's used for the processing. Once it's not used, the input layer should be removed from the internal storage. This should also be applied to the scenario outputs.
• Worker concurrency management. The number of concurrency in the worker will determine how many task that the worker can run at the same time. We should consider the resource that a worker may have (memory, internal storage) before deciding on the number of the concurrency for a worker.
• Worker scalling. We can explore on how to scale the worker containers automatically when there are queued tasks and the existing container is still busy processing current tasks.

[danangmassandy]

I have tested running the plugin code using PyQGIS in a worker container with a number of scenarios. I compared the performance with QGIS desktop (v3.28.1) running on my windows machine. Below are the performance comparison:	Scenario	Qgis (in seconds)	Worker (in seconds)	Worker max memory (in GB)
Scenario 1	5.73	2.85	1.7
Scenario 2	9.41	3.11	1.8
Scenario 3	10.38	3.4	1.8
Scenario 4	15.53	4.94	1.8
Scenario 5	25.43	6.82	1.8
Scenario 6	88.22	33.08	2.3

Scenarios:

• Scenario 1: 1 implementation model with 1 pathway: Alient Plant Removal
• Scenario 2: 1 implementation model with 1 pathway: Avoided Wetland Conversion
• Scenario 3:1 implementation model with 1 pathway: Woody Encroachment Control
• Scenario 4: 1 implementation model (Alien Plant Removal) with 2 pathways: Alient Plant Removal + Avoided Wetland Conversion
• Scenario 5: 1 implementation model (Alien Plant Removal) with 3 pathways
• Scenario 6: 3 implementation models, 1) Alien Plant Removal with 3 pathways 2) Avoided Wetland Conversion/Restoration with 2 pathways 3) Natural Woodland Livestock Management with 2 pathways

Note: before running each scenario, QGIS/worker container is restarted.