Batch Processing Freezes

[Mikuohatsune53]

What is the bug or the crash?

When running a simple batch process with only 12 rows, it freezes up and does not continue it. Even after waiting for over an hour, it is still frozen. Even after clicking the cancel button, it freezes up unless I click entirely out of QGis.

Sadly no error codes, but the issue appears after running a batch process even on another computer. My friend also tried the same process on his computer with a different set of temp data, and he also ran into the same issue.

Steps to reproduce the issue

Join Attributes by Location (summary) --> Batch Processing --> County Intersect TMax

Versions

The current released version of QGIS is 3.32.0.

Supported QGIS version

• [X] I'm running a supported QGIS version according to the roadmap.

New profile

• [X] I tried with a new QGIS profile

Additional context

No response

[Mikuohatsune53]

Version:

QGIS version 3.32.0-Lima QGIS code revision 311a8cb8a6 Qt version 5.15.3 Python version 3.9.5 GDAL/OGR version 3.7.0 PROJ version 9.2.1 EPSG Registry database version v10.088 (2023-05-13) GEOS version 3.11.2-CAPI-1.17.2 SQLite version 3.41.1 PDAL version 2.5.3 PostgreSQL client version 15.2 SpatiaLite version 5.0.1 QWT version 6.1.6 QScintilla2 version 2.13.1 OS version Windows 10 Version 2009

Active Python plugins quick_map_services 0.19.29 raster_cutter 0.6 db_manager 0.1.20 grassprovider 2.12.99 MetaSearch 0.3.6 otbprovider 2.12.99 processing 2.12.99

[roya0045]

As a workaround, you can try to close qgis, a popup will open, click yes to cancel pending tasks, qgis should remain open, hit run again and it may finish this time.

[Mides]

The same problem occurs in 3.28.9.

[wikirikitiki]

Unfortunately I have the same problem. QGIS 3.24 is the last version that has no problem with batch processing. Everything after (3.26 up to 3.32) and the batch processing freezes without crashing.

I tried with with a new QGIS profile but without success. I also tried the same with 3 other computers at my workplace (all have different specs, but same OS - Windows 10 Pro 20H2).

It doesn't matter how many "rows" are there, how big are the files or how resource heavy is process being used - the batch processing freezes anyway.

[nyalldawson]

I've tried extensively to reproduce this with no luck.

If it's still reproducible in current QGIS versions, can someone please post a workflow which is able to demonstrate the issue?

[Mides]

If you select layers loaded in Qgis, the problem occurs. If you load layers from a file, the problem no longer exists.

[nicogodet]

@nyalldawson It's very hard (impossible?) to provide a sample project to reproduce... This deadlock is very random. You can try Steps to reproduce listed here: https://github.com/qgis/QGIS/issues/54124 I do not garante it will happen (and it might be a Windows issue too) but when I wrote #54124, I was able to reproduce it everytime.

[roya0045]

@nyalldawson It's very hard (impossible?) to provide a sample project to reproduce... This deadlock is very random. You can try Steps to reproduce listed here: #54124 I do not garante it will happen (and it might be a Windows issue too) but when I wrote #54124, I was able to reproduce it everytime.

It also seems to happen more commonly when there are a lot of items in the batch (40+) and will stop at a random point. Using gdal translate to change tiff to jpg in batch is also an easy case. Just split a tiff into many subfile (I get mine fron GEE though), then change their extension with gdal translate. With 120 small files I get a stall around 20 to 40 items done-> Trigger the batch cancel-> keep the dialog open-> remove the first elements that were processed -> run again -> repeat until all done.

[wikirikitiki]

Sorry for replying so late. I'm not a dev so I have no idea it this is what you wanted Nyall. Please let me know if there's anything else I can do.

I've uploaded two videos to a Google Drive folder here

Both videos show QGIS version 3.32.2 on a Windows 10 machine (the os is up to date).

First video is called workflow_1.mp4. It shows batch processing of "Clip raster by mask layer". I'm using 15 rasters (small file sizes) and 15 polygons to mask each raster with corresponding polygon. The video shows that batch processing got stuck at random moment and I had to cancel the batch processing and rerun it. Windows task manager shows that QGIS is fine, only the batch processing got stuck. Fortunately I was able to rerun it successfully the second time (which happens very rarely).

Second video is called workflow_2.mp4. It shows batch processing of "Clip". I'm clipping one geopackage containing a lot of points and clipping that one file using multiple polygons. The video shows batch processing getting stuck multiple times at random moments.

I've tried the same today on QGIS 3.32.3 but the problem still occurs. This problem does not appear on QGIS 3.24 and earlier.

[nyalldawson]

Unfortunately I still can't reproduce . I'll have to leave this to someone else to debug.

[GTP85]

Hello, I have exactly the same problem on various batch processing models, when I try to work on 70, 100, or 800 files.

Thanks for the solution to use QGIS 3.24 (3.24.3 for me), it works fine with that version!

QGIS 3.32.3 (but same problem with an older version). Windows 10 Pro Intel(R) Core(TM) i5-10310U CPU @ 1.70GHz 2.21 GHz

[nicogodet]

@nyalldawson It might be a windows only @roya0045 it happens on Linux for you?

[GTP85]

If you select layers loaded in Qgis, the problem occurs. If you load layers from a file, the problem no longer exists.

For me, the problem occurs with files (not loaded as QGIS layers). I didn't tried with layers already loaded in QGIS.

[roya0045]

@nyalldawson It might be a windows only @roya0045 it happens on Linux for you?

windows, so I assume it's wondows blocking too many subprocess or something to that effect.

[nyalldawson]

Can someone please test a build including https://github.com/qgis/QGIS/pull/54845 and report whether the issue is fixed?

[GTP85]

Can someone please test a build including #54845 and report whether the issue is fixed?

I have no idea how to work with builds, but if someone can send me a "how-to", I could test that.

[nicogodet]

Download this artifact https://github.com/qgis/QGIS/suites/17011932137/artifacts/971564009

Extract twice and run qgis.exe in bin folder

[nicogodet]

@nyalldawson Same issue with #54845

And gdalwarp was not even run.

[baswein]

Thanks for trying @nyalldawson! It is still a problem in todays QGIS 3.35.00 from osgeo4w. I find it is hit or miss I have been running a bunch of batch processing with 90 layers over the last couple of months. I think less than 10% of the time it gets all the way through. Often it stalls multiple times during the processing. This is of course very annoying as I can't leave thigs to process over night and have to keep checking to see if it is stalled out. Is there anyway I can log what is happening to try to get to the bottom of it?

[Eksploracja]

According to this StackExchange thread, it seems that the aforementioned issue does not occur in the Ubuntu system https://gis.stackexchange.com/questions/464309/qgis-batch-processing-stops-part-way-through-indefinitely-for-no-apparent-reason

[sivansakthi]

Hello @agiudiceandrea,

I have done many times re-projecting the layers. Please refer the attached recorded videos as zip file 2024-04-01-17-33-38.zip

Please have a look and let me know what to do.

Regards, Sivansakthi

[effjot]

Happens with version 3.34.6 on Windows 10, too. A few layers are ok, but with more than ≈10, processing randomly gets stuck.

[rowan8k]

Having the same issue on 3.28.13 on Windows 11. Trying to do a batch process with 10 rows, gets stuck on 3-5.

[baswein]

Having the same issue on 3.28.13 on Windows 11. Trying to do a batch process with 10 rows, gets stuck on 3-5.

Does it happen every time? Just wondering because that is a relatively small number of processes. If we can find a dataset that is consistent then someone might be able to track down this very annoying issue.

[rowan8k]

Having the same issue on 3.28.13 on Windows 11. Trying to do a batch process with 10 rows, gets stuck on 3-5.

Does it happen every time? Just wondering because that is a relatively small number of processes. If we can find a dataset that is consistent then someone might be able to track down this very annoying issue.

The 3 times I tried it, it kept happening. After that I switched to linux and it ran flawlessly with 10 rows and 300 rows. I can share the data if you want.

[baswein]

Please send it my way I'll try it on my end and see if there is any consistency.

[rowan8k]

Find the attached zip with all the data.

This analysis was done in a graphical model which I also included. I ran the graphical model as a batch process with the following inputs:

• End points: green_area_points
• Network: ways_kaukajarvi
• Start points: Add values by expression with the expression: "aggregate('buildings_kaukajarvi_points','array_agg',$geometry)"

Output was saved as "shortest_path.gpkg" with autofill mode set to fill with numbers. data.zip

[tsmcgrath]

@nyalldawson It might be a windows only @roya0045 it happens on Linux for you?

It's happening on my Mac M1 (ARM Native build 3.36) https://github.com/opengisch/qgis-conda-builder/releases Report when it was Forcibly Quit: 20240914_FreezingStackTrace.txt

Don't have a repro yet. Seems random. It's not freezing at the same place and is progressive. Here's my last three logs of the run. The first was a full, successful run of 10. I ended up forcing the quit on the last two. I'm going to try it with 3.38 (ARM native) and see if it continues and work on a repro package of data and code.

2024-09-14T06-53-08.log 2024-09-14T06-24-14.log 2024-09-14T06-19-11.log

[tsmcgrath]

Here's a repro case. Repro steps in the README.md. Repro's on a Mac M1 with a native ARM build (3.36 & 3.38) Bug20240914_Freeze.zip

[baswein]

@rowan8k 's data.zip algo stopped when trying to run the 7th file then on subsequent restarts the 7th,3rd,10th,10th, and 8th files I stopped after that. This is more frequent than I normally experience and might therefor be useful for someone who knew how to see a log of what the hang up might be (not me). I did try @tsmcgrath 's python script but couldn't get it to run right away.

[tsmcgrath]

@baswein What was the error? I can go through the repro steps again and see if I missed something in the setup.

[baswein]

@baswein What was the error? I can go through the repro steps again and see if I missed something in the setup.

I got an out of index error on line 37 - layer = QgsProject.instance().mapLayersByName(layername)[0] I assume this means that it couldn't load the layer?

[tsmcgrath]

Hi @baswein, QGIS sometimes adds new strings when you add a layer and when I built the repro case it did this in the project. Just rename the layer to be 'CatchSanSaba_v1-6_3083' and it will match the variable: layername = 'CatchSanSaba_v1-6_3083' Or just rename both to make them match.

Latest crash report: crashreport_20240929.txt

[nyalldawson]

Can someone test if the PR at https://github.com/qgis/QGIS/pull/58337 fixes this?

[nicogodet]

@nyalldawson I get the same freeze

Trying to get a GDB output but it kept crashing at random point before I run alg...

[github-actions[bot]]

The QGIS project highly values your report and would love to see it addressed. However, this issue has been left in feedback mode for the last 14 days and is being automatically marked as "stale". If you would like to continue with this issue, please provide any missing information or answer any open questions. If you could resolve the issue yourself meanwhile, please leave a note for future readers with the same problem and close the issue. In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this issue. If there is no further activity on this issue, it will be closed in a week.

[tsmcgrath]

I can confirm this is definitely still happening for me. My repro case above still repro's 100% of the time (when the layername variable matches the layer name). It also happens for other datasets. Definitely seems data driven because it does not occur with other data instances.

[nicogodet]

@tsmcgrath I don't understand

Repro Steps

• Start QGIS (repro's in 3.36 ARM Native release and 3.38)
• Open Project FreezeRepro.qgz
• Open Layout Manager.
• Open ReferenceMap layout.
• In both mapconfig.py and mapgen_reference.py code change "repro_path" path to your local path
• Open Python console
• Open the Editor (Show Editor button)
• Paste the mapgen_reference.py code into the editor

There is nothing in above stepts related to a batch processing freeze, you execute a script not a batch processing...

[nicogodet]

bug_batch.json

1. Load batch json in create raster with gamma distrib
2. Adapt output path with this expression : '<valid_path_depeing_on_your_setup>' || @row_number ||'.tif'

Execute

I execute above steps twice : first freeze à 7th row, second at 18th. Completly random

I can still reproduce in Win 3.38.3 but not in Win Qt6 master 699c004a5d

[tsmcgrath]

@nicogodet focus on the freezing, not the batch. Plus I run 10, 100, X, maps iterating with that script. How is that not "batch"? It freezes after just a couple. Also, I'm not sure it's completely random on my Mac M1. Pretty sure it freezes consistently as it is iterating on one of them. Which may be useful in a repro case.