Rasterize mesh dataset tool: python error when running it in batch mode

[Brent-Edwards]

When running the said tool in batch mode (Right click>Execute as Batch Process) with several hundred processes (100s of lines), the tool crashes intermittently after successfully generating several, but not all of the output rasters. The batch processes are populated by a JSON file via the Open (Batch) button in the tool dialog. I am able to rerun the batch iteratively starting from the last successfully generated output raster and all of the processes execute successfully.

Here's a sample of the JSON batch file:

[{"PARAMETERS": {"INPUT": "'//path/to/my/mesh.2dm'", "DATASET_GROUPS": "[0]", "DATASET_TIME": "{'type': 'static'}", "EXTENT": "'300000, 301000, 5038000, 5039000'", "PIXEL_SIZE": "0.25", "CRS_OUTPUT": "QgsCoordinateReferenceSystem('EPSG:2950')"}, "OUTPUTS": {"OUTPUT": "//path/to/my/output/rasters/tuile_300-5038.tif"}}, {"PARAMETERS": {"INPUT": "'//path/to/my/mesh.2dm'", "DATASET_GROUPS": "[0]", "DATASET_TIME": "{'type': 'static'}", "EXTENT": "'300000, 301000, 5039000, 5040000'", "PIXEL_SIZE": "0.25", "CRS_OUTPUT": "QgsCoordinateReferenceSystem('EPSG:2950')"}, "OUTPUTS": {"OUTPUT": "//path/to/my/output/rasters/tuile_300-5039.tif"}}, {"PARAMETERS": {"INPUT": "'//path/to/my/mesh.2dm'", "DATASET_GROUPS": "[0]", "DATASET_TIME": "{'type': 'static'}", "EXTENT": "'300000, 301000, 5040000, 5041000'", "PIXEL_SIZE": "0.25", "CRS_OUTPUT": "QgsCoordinateReferenceSystem('EPSG:2950')"}, "OUTPUTS": {"OUTPUT": "//path/to/my/output/rasters/tuile_300-5040.tif"}}]

I am also able to run the same entire batch process successfully with the old Crayfish plugin version of the tool on QGIS 3.10 without errors/crashes.

Error message:

`An error has occurred while executing Python code:

Exception: unknown Traceback (most recent call last): File "C:/QGIS_L~1/apps/qgis/./python/plugins\processing\gui\BatchAlgorithmDialog.py", line 157, in runAlgorithm results, ok = self.algorithm().run(parameters, context, multi_feedback) Exception: unknown

Python version: 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] QGIS version: 3.18.0-Zürich Zürich, bdef9fb328

Python Path: C:/Users/mywinusername/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LAStools C:/QGIS_L~1/apps/qgis/./python C:/Users/mywinusername/AppData/Roaming/QGIS/QGIS3\profiles\default/python C:/Users/mywinusername/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins C:/QGIS_L~1/apps/qgis/./python/plugins C:\QGIS_L~1\apps\Python37 C:\QGIS_L~1\apps\Python37\Scripts C:\QGIS_latest\bin\python37.zip C:\QGIS_L~1\apps\Python37\DLLs C:\QGIS_L~1\apps\Python37\lib C:\QGIS_latest\bin C:\QGIS_L~1\apps\Python37\lib\site-packages C:\QGIS_L~1\apps\Python37\lib\site-packages\win32 C:\QGIS_L~1\apps\Python37\lib\site-packages\win32\lib C:\QGIS_L~1\apps\Python37\lib\site-packages\Pythonwin C:/Users/mywinusername/AppData/Roaming/QGIS/QGIS3\profiles\default/python C:\Users\mywinusername\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\crayfish\gui..\pyqtgraph-0.10.0-py2.py3-none-any.whl C:\Users\mywinusername\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\QuickMultiAttributeEdit3/forms`

QGIS version info:

QGIS version 3.18.0-Zürich QGIS code revision bdef9fb328 Compiled against Qt 5.11.2 Running against Qt 5.11.2 Compiled against GDAL/OGR 3.1.4 Running against GDAL/OGR 3.1.4 Compiled against GEOS 3.8.1-CAPI-1.13.3 Running against GEOS 3.8.1-CAPI-1.13.3 Compiled against SQLite 3.29.0 Running against SQLite 3.29.0 PostgreSQL Client Version 11.5 SpatiaLite Version 4.3.0 QWT Version 6.1.3 QScintilla2 Version 2.10.8 Compiled against PROJ 6.3.2 Running against PROJ Rel. 6.3.2, May 1st, 2020 OS Version Windows 10 (10.0) Active python plugins changeDataSource; crayfish; ImportPhotos; LAStools; latlontools; LecoS; Mergin; profiletool; qgis-maptiler-plugin; QPackage; QuickMultiAttributeEdit3; quick_map_services; splitmultipart; valuetool; db_manager; processing

[vcloarec]

@Brent-Edwards I had made some tries and it works well with 3.18.2. Can you try with the last 3.18?

[Brent-Edwards]

@Brent-Edwards I had made some tried and it works well with 3.18.2. Can you try with the last 3.18?

Thanks for the info. I'll give it a try with the latest release of 3.18.* and report back.

[github-actions[bot]]

The QGIS project highly values your report and would love to see it addressed. However, this issue has been left in feedback mode for the last 14 days and is being automatically marked as "stale". If you would like to continue with this issue, please provide any missing information or answer any open questions. If you could resolve the issue yourself meanwhile, please leave a note for future readers with the same problem and close the issue. In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this issue. If there is no further activity on this issue, it will be closed in a week.

[mattfack]

I am having a similar problem with qgis 3.18.3:

I generate a json file with the "run as batch" dialog, then try to load it and I get the python crash as in the present issue description.

[vcloarec]

@mattfack , is it possible to share data which for there is this issue? ideally a project with corresponding files

[mattfack]

I work with sensitive data, unfortunately I cannot share them.

But in my case the error is pretty straightforward:

NameError: name 'QgsMeshDatasetIndex' is not defined Traceback (most recent call last): File "C:/PROGRA~1/QGIS3~1.18/apps/qgis/./python/plugins\processing\gui\BatchPanel.py", line 553, in load value = eval(params[param.name()]) File "", line 1, in NameError: name 'QgsMeshDatasetIndex' is not defined

[vcloarec]

@mattfack , your error is not exactly the same. I think your error happens when you use the Dataset time parameter with "Dataset group time step" option, no ? I think I have reproduced it under Windows platform.

[mattfack]

Yes, I noticed that. However, if I switch to "Defined date/time" I can save and import the file but the "Dataset time" will be imported incorrectly (and set to the default "Dataset grup timeset")

[vcloarec]

@mattfack Indeed, there are some other bad issues... This PR https://github.com/qgis/QGIS/pull/43565 should fix them

[vcloarec]

@mattfack , the PR was merged in master, don't hesitate to test with master if you can.

[github-actions[bot]]

The QGIS project highly values your report and would love to see it addressed. However, this issue has been left in feedback mode for the last 14 days and is being automatically marked as "stale". If you would like to continue with this issue, please provide any missing information or answer any open questions. If you could resolve the issue yourself meanwhile, please leave a note for future readers with the same problem and close the issue. In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this issue. If there is no further activity on this issue, it will be closed in a week.

[github-actions[bot]]

While we hate to see this happen, this issue has been automatically closed because it has not had any activity in the last 42 days despite being marked as feedback. If this issue should be reconsidered, please follow the guidelines in the previous comment and reopen this issue. Or, if you have any further questions, there are also further support channels that can help you.

[Brent-Edwards]

Just tested with QGIS 3.22 and the issue is still present. The process crashes after successfully executing 143 lines of my 187 line batch JSON command file. The second run on the remaining 45 lines executed successfully without a crash.

FYI, my batch file rasterizes the mesh in 1km x 1km tiles at a resolution of 25cm - 122MB per tile.

[vcloarec]

hmmm, I have to try on my side with the same workflow... In the meanwhile, even I don't see any reason, I am wondering if it is a RAM issue... what is your OS and memory capacity ? have you memory the memory usage during the process?

[Brent-Edwards]

It could indeed be memory related. This is really the only tool that I run using the built-in "Execute as Batch Process" functionality so it is hard for me to provide any useful comments beyond the obvious thing to look at would be to ensure that the process is indeed running a single line/command at a time or multiprocessing within the limits of the available RAM/cores.

Here are some stats on the input 2mb 2DM mesh that I'm rasterizing:

Extent 265039.2999999999883585,4928886.9900000002235174 : 352513.0800000000162981,5112421.5400000000372529 Vertex count 26 630 Face count 53 072 Edge count 0 Dataset groups count 1

As a side note, it is quite slow to load the JSON file containing the commands once the number of lines/commands grows into the 100s. I tried one today containing over 4000 lines and I crashed out of QGIS after waiting more than 20 minutes for the JSON batch file to load.

As noted previously, I tested a similar batch that failed with processing version of the "Rasterize mesh dataset" tool, but worked in a previous Crayfish version of the tool.

Here's my system info:

[vcloarec]

Got it, indeed, crash due to memory consumption coming from a memory leak.

Should be fixed by #45890 and backported to the next 3.22