Batch Processing Freezes

Mikuohatsune53 commented 11 months ago

What is the bug or the crash?

When running a simple batch process with only 12 rows, it freezes up and does not continue it. Even after waiting for over an hour, it is still frozen. Even after clicking the cancel button, it freezes up unless I click entirely out of QGis.

Sadly no error codes, but the issue appears after running a batch process even on another computer. My friend also tried the same process on his computer with a different set of temp data, and he also ran into the same issue.

Steps to reproduce the issue

Join Attributes by Location (summary) --> Batch Processing --> County Intersect TMax

Versions

The current released version of QGIS is 3.32.0.

Supported QGIS version

[X] I'm running a supported QGIS version according to the roadmap.

New profile

[X] I tried with a new QGIS profile

Additional context

No response

Mikuohatsune53 commented 11 months ago

Version:

QGIS version 3.32.0-Lima QGIS code revision 311a8cb8a6 Qt version 5.15.3 Python version 3.9.5 GDAL/OGR version 3.7.0 PROJ version 9.2.1 EPSG Registry database version v10.088 (2023-05-13) GEOS version 3.11.2-CAPI-1.17.2 SQLite version 3.41.1 PDAL version 2.5.3 PostgreSQL client version 15.2 SpatiaLite version 5.0.1 QWT version 6.1.6 QScintilla2 version 2.13.1 OS version Windows 10 Version 2009

Active Python plugins quick_map_services 0.19.29 raster_cutter 0.6 db_manager 0.1.20 grassprovider 2.12.99 MetaSearch 0.3.6 otbprovider 2.12.99 processing 2.12.99

roya0045 commented 11 months ago

As a workaround, you can try to close qgis, a popup will open, click yes to cancel pending tasks, qgis should remain open, hit run again and it may finish this time.

Mides commented 11 months ago

The same problem occurs in 3.28.9.

wikirikitiki commented 10 months ago

Unfortunately I have the same problem. QGIS 3.24 is the last version that has no problem with batch processing. Everything after (3.26 up to 3.32) and the batch processing freezes without crashing.

I tried with with a new QGIS profile but without success. I also tried the same with 3 other computers at my workplace (all have different specs, but same OS - Windows 10 Pro 20H2).

It doesn't matter how many "rows" are there, how big are the files or how resource heavy is process being used - the batch processing freezes anyway.

nyalldawson commented 9 months ago

I've tried extensively to reproduce this with no luck.

If it's still reproducible in current QGIS versions, can someone please post a workflow which is able to demonstrate the issue?

Mides commented 9 months ago

If you select layers loaded in Qgis, the problem occurs. If you load layers from a file, the problem no longer exists.

nicogodet commented 9 months ago

@nyalldawson It's very hard (impossible?) to provide a sample project to reproduce... This deadlock is very random. You can try Steps to reproduce listed here: https://github.com/qgis/QGIS/issues/54124 I do not garante it will happen (and it might be a Windows issue too) but when I wrote #54124, I was able to reproduce it everytime.

roya0045 commented 9 months ago

@nyalldawson It's very hard (impossible?) to provide a sample project to reproduce... This deadlock is very random. You can try Steps to reproduce listed here: #54124 I do not garante it will happen (and it might be a Windows issue too) but when I wrote #54124, I was able to reproduce it everytime.

It also seems to happen more commonly when there are a lot of items in the batch (40+) and will stop at a random point. Using gdal translate to change tiff to jpg in batch is also an easy case. Just split a tiff into many subfile (I get mine fron GEE though), then change their extension with gdal translate. With 120 small files I get a stall around 20 to 40 items done-> Trigger the batch cancel-> keep the dialog open-> remove the first elements that were processed -> run again -> repeat until all done.

wikirikitiki commented 9 months ago

Sorry for replying so late. I'm not a dev so I have no idea it this is what you wanted Nyall. Please let me know if there's anything else I can do.

I've uploaded two videos to a Google Drive folder here

Both videos show QGIS version 3.32.2 on a Windows 10 machine (the os is up to date).

First video is called workflow_1.mp4. It shows batch processing of "Clip raster by mask layer". I'm using 15 rasters (small file sizes) and 15 polygons to mask each raster with corresponding polygon. The video shows that batch processing got stuck at random moment and I had to cancel the batch processing and rerun it. Windows task manager shows that QGIS is fine, only the batch processing got stuck. Fortunately I was able to rerun it successfully the second time (which happens very rarely).

Second video is called workflow_2.mp4. It shows batch processing of "Clip". I'm clipping one geopackage containing a lot of points and clipping that one file using multiple polygons. The video shows batch processing getting stuck multiple times at random moments.

I've tried the same today on QGIS 3.32.3 but the problem still occurs. This problem does not appear on QGIS 3.24 and earlier.

nyalldawson commented 8 months ago

Unfortunately I still can't reproduce . I'll have to leave this to someone else to debug.

GTP85 commented 8 months ago

Hello, I have exactly the same problem on various batch processing models, when I try to work on 70, 100, or 800 files.

Thanks for the solution to use QGIS 3.24 (3.24.3 for me), it works fine with that version!

QGIS 3.32.3 (but same problem with an older version). Windows 10 Pro Intel(R) Core(TM) i5-10310U CPU @ 1.70GHz 2.21 GHz

nicogodet commented 8 months ago

@nyalldawson It might be a windows only @roya0045 it happens on Linux for you?

GTP85 commented 8 months ago

If you select layers loaded in Qgis, the problem occurs. If you load layers from a file, the problem no longer exists.

For me, the problem occurs with files (not loaded as QGIS layers). I didn't tried with layers already loaded in QGIS.

roya0045 commented 8 months ago

@nyalldawson It might be a windows only @roya0045 it happens on Linux for you?

windows, so I assume it's wondows blocking too many subprocess or something to that effect.

nyalldawson commented 8 months ago

Can someone please test a build including https://github.com/qgis/QGIS/pull/54845 and report whether the issue is fixed?

GTP85 commented 8 months ago

Can someone please test a build including #54845 and report whether the issue is fixed?

I have no idea how to work with builds, but if someone can send me a "how-to", I could test that.

nicogodet commented 8 months ago

Download this artifact https://github.com/qgis/QGIS/suites/17011932137/artifacts/971564009

Extract twice and run qgis.exe in bin folder

nicogodet commented 8 months ago

@nyalldawson Same issue with #54845

And gdalwarp was not even run.

baswein commented 7 months ago

Thanks for trying @nyalldawson! It is still a problem in todays QGIS 3.35.00 from osgeo4w. I find it is hit or miss I have been running a bunch of batch processing with 90 layers over the last couple of months. I think less than 10% of the time it gets all the way through. Often it stalls multiple times during the processing. This is of course very annoying as I can't leave thigs to process over night and have to keep checking to see if it is stalled out. Is there anyway I can log what is happening to try to get to the bottom of it?