search

qgis / QGIS

QGIS is a free, open source, cross platform (lin/win/mac) geographical information system (GIS)
https://qgis.org
GNU General Public License v2.0
10.46k stars 2.99k forks source link

The sytem crashes when trying to extract a raster by using layer by extent . Now it also failed for extract by mask. after four #56871

Open njc888 opened 6 months ago

njc888 commented 6 months ago

What is the bug or the crash?

the system crashes with extraction from a virtual raster built from 4 tiff files by using extract by layer extent and layer mask

Steps to reproduce the issue

User Feedback

i was running extract by layer mask and performing it on a virtual raster built from four rasters this is the fourth occurence crashing without success after 2 hrs each

Report Details

Python Stack Trace

Windows fatal exception: access violation

Thread 0x00000a2c (most recent call first):
<no Python frame>

Stack Trace


QObject::thread :
QgsProcessingAlgorithm::runPrepared :
QgsProcessingAlgRunnerTask::run :
pdal::Option::getName :
QgsTask::start :
QThreadPoolPrivate::reset :
QThread::start :
BaseThreadInitThunk :
RtlUserThreadStart :

QGIS Info QGIS Version: 3.34.3-Prizren QGIS code revision: 47373234ac Compiled against Qt: 5.15.3 Running against Qt: 5.15.3 Compiled against GDAL: 3.8.3 Running against GDAL: 3.8.3

System Info CPU Type: x86_64 Kernel Type: winnt Kernel Version: 10.0.22631

Versions

QGIS version 3.34.3-Prizren QGIS code revision 47373234ac Qt version 5.15.3 Python version 3.9.18 GDAL/OGR version 3.8.3 PROJ version 9.3.1 EPSG Registry database version v10.098 (2023-11-24) GEOS version 3.12.1-CAPI-1.18.1 SQLite version 3.41.1 PDAL version 2.6.0 PostgreSQL client version 15.2 SpatiaLite version 5.1.0 QWT version 6.1.6 QScintilla2 version 2.13.4 OS version Windows 10 Version 2009

Active Python plugins cesium_ion v1.0.2 clipper 1.2 CloudMasking 23.3.30 Coordtransform 1.1 crayfish 3.6.0 DEMto3D 3.6 HistMatch-master 0.2.0 map-units-per-pixel 0.2.1 processing_saga_nextgen 1.0.0 QuickOSM 2.2.3 quick_map_services 0.19.34 valuetool 3.0.19 VectorBender 0.2.1 grassprovider 2.12.99 otbprovider 2.12.99 processing 2.12.99

Supported QGIS version

New profile

Additional context

No response

njc888 commented 6 months ago

the bug appears randomly. Now after submitting report it completed the said protocal in 87 seconds QGIS version: 3.34.3-Prizren QGIS code revision: 47373234ac Qt version: 5.15.3 Python version: 3.9.18 GDAL version: 3.8.3 GEOS version: 3.12.1-CAPI-1.18.1 PROJ version: Rel. 9.3.1, December 1st, 2023 PDAL version: 2.6.0 (git-version: 3fced5) Algorithm started at: 2024-03-15T16:50:29 Algorithm 'Clip raster by mask layer' starting… Input parameters: { 'ALPHA_BAND' : False, 'CROP_TO_CUTLINE' : True, 'DATA_TYPE' : 0, 'EXTRA' : '', 'INPUT' : 'C:/Users/Power/AppData/Local/Temp/processing_HcXWbY/732360bd78fb4fadbd239bcf9d5fefcb/OUTPUT.vrt', 'KEEP_RESOLUTION' : True, 'MASK' : 'C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/BoundingConvexTvmBorderDist.shp', 'MULTITHREADING' : False, 'NODATA' : None, 'OPTIONS' : '', 'OUTPUT' : 'C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/demtvmBorderdist.tif', 'SET_RESOLUTION' : False, 'SOURCE_CRS' : None, 'TARGET_CRS' : None, 'TARGET_EXTENT' : None, 'X_RESOLUTION' : None, 'Y_RESOLUTION' : None }

GDAL command: gdalwarp -overwrite -of GTiff -tr 0.0002777777777777778 -0.0002777777777777778 -tap -cutline "C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/BoundingConvexTvmBorderDist.shp" -cl BoundingConvexTvmBorderDist -crop_to_cutline C:/Users/Power/AppData/Local/Temp/processing_HcXWbY/732360bd78fb4fadbd239bcf9d5fefcb/OUTPUT.vrt "C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/demtvmBorderdist.tif" GDAL command output: ERROR 4: Unable to open C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/demtvmBorderdist.tif to obtain file list. Creating output file that is 4680P x 4834L. Processing C:/Users/Power/AppData/Local/Temp/processing_HcXWbY/732360bd78fb4fadbd239bcf9d5fefcb/OUTPUT.vrt [1/1] : 0...10...20...30...40...50...60...70...80...90...100 - done. Process completed successfully Execution completed in 0.87 seconds Results: {'OUTPUT': 'C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful ' 'layers/New Data/Tn Distrcts Tvm Border/demtvmBorderdist.tif'}

Loading resulting layers Algorithm 'Clip raster by mask layer' finished

the previous attempts are

processing.run("gdal:cliprasterbymasklayer", {'INPUT':'C:/Users/Power/AppData/Local/Temp/processing_HcXWbY/732360bd78fb4fadbd239bcf9d5fefcb/OUTPUT.vrt','MASK':'C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/Border Districts.shp','SOURCE_CRS':QgsCoordinateReferenceSystem('EPSG:4326'),'TARGET_CRS':QgsCoordinateReferenceSystem('EPSG:32643'),'TARGET_EXTENT':None,'NODATA':None,'ALPHA_BAND':False,'CROP_TO_CUTLINE':True,'KEEP_RESOLUTION':True,'SET_RESOLUTION':False,'X_RESOLUTION':None,'Y_RESOLUTION':None,'MULTITHREADING':True,'OPTIONS':'COMPRESS=DEFLATE|PREDICTOR=2|ZLEVEL=9','DATA_TYPE':0,'EXTRA':'','OUTPUT':'C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/demtvmBorderdist.tif'})

the process failed for the command above

however the command sent this time gdalwarp -overwrite -of GTiff -tr 0.0002777777777777778 -0.0002777777777777778 -tap -cutline "C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/BoundingConvexTvmBorderDist.shp" -cl BoundingConvexTvmBorderDist -crop_to_cutline C:/Users/Power/AppData/Local/Temp/processing_HcXWbY/732360bd78fb4fadbd239bcf9d5fefcb/OUTPUT.vrt "C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/demtvmBorderdist.tif"

worked

agiudiceandrea commented 6 months ago

@njc888, thanks for reporting. It would be useful if you provided the layer / parameters / exact steps with which the issue occurs.

njc888 commented 6 months ago

these are tiles i used next i built a virtual raster in memory then i tried to extract by using layer extent , the command passed was

gdal_translate -projwin 3628750.0929 2403875.6989 3773091.1086 2252838.0288 -of GTiff C:/Users/Power/AppData/Local/Temp/processing_HcXWbY/732360bd78fb4fadbd239bcf9d5fefcb/OUTPUT.vrt C:/Users/Power/AppData/Local/Temp/processing_AjGEna/d2b9358ec3f444b1bc7ba99d535d4b43/OUTPUT.tif

the log output was

GDAL command output: ERROR 3: OUTPUT.tif: Free disk space available is 793997127680 bytes, whereas 565080124182042176 are at least necessary. You can disable this check by defining the CHECK_DISK_FREE_SPACE configuration option to FALSE. Input file size is 7200, 7200 Process returned error code 1 Execution completed in 0.23 seconds Results: {'OUTPUT': 'C:/Users/Power/AppData/Local/Temp/processing_AjGEna/e332bdd362af4fa5a5c18663bc1cb889/OUTPUT.tif'}

Loading resulting layers The following layers were not correctly generated. • C:/Users/Power/AppData/Local/Temp/processing_AjGEna/e332bdd362af4fa5a5c18663bc1cb889/OUTPUT.tif You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.

Next i tried by giving a compression and the parameters were changed to the following command and the output is as follows QGIS version: 3.34.3-Prizren QGIS code revision: 47373234ac Qt version: 5.15.3 Python version: 3.9.18 GDAL version: 3.8.3 GEOS version: 3.12.1-CAPI-1.18.1 PROJ version: Rel. 9.3.1, December 1st, 2023 PDAL version: 2.6.0 (git-version: 3fced5) Algorithm started at: 2024-03-17T20:07:02 Algorithm 'Clip raster by extent' starting… Input parameters: { 'DATA_TYPE' : 0, 'EXTRA' : '', 'INPUT' : 'C:/Users/Power/AppData/Local/Temp/processing_HcXWbY/732360bd78fb4fadbd239bcf9d5fefcb/OUTPUT.vrt', 'NODATA' : None, 'OPTIONS' : 'COMPRESS=DEFLATE|PREDICTOR=2|ZLEVEL=9', 'OUTPUT' : 'TEMPORARY_OUTPUT', 'OVERCRS' : False, 'PROJWIN' : '3628750.092900000,3773091.108600000,2252838.028800000,2403875.698900000 []' }

GDAL command: gdal_translate -projwin 3628750.0929 2403875.6989 3773091.1086 2252838.0288 -of GTiff -co COMPRESS=DEFLATE -co PREDICTOR=2 -co ZLEVEL=9 C:/Users/Power/AppData/Local/Temp/processing_HcXWbY/732360bd78fb4fadbd239bcf9d5fefcb/OUTPUT.vrt C:/Users/Power/AppData/Local/Temp/processing_AjGEna/46f1679c887641ba8e8f177bfacff9df/OUTPUT.tif GDAL command output: ERROR 1: TIFFSetupStrips:Too large Strip/Tile Offsets/ByteCounts arrays ERROR 1: GTiffCreateCopy():No space for strip arrays ERROR 1: MissingRequired:TIFF directory is missing required "StripOffsets" field ERROR 1: MissingRequired:TIFF directory is missing required "StripOffsets" field Input file size is 7200, 7200 Process returned error code 1 Execution completed in 0.11 seconds Results: {'OUTPUT': 'C:/Users/Power/AppData/Local/Temp/processing_AjGEna/46f1679c887641ba8e8f177bfacff9df/OUTPUT.tif'}

Loading resulting layers The following layers were not correctly generated. • C:/Users/Power/AppData/Local/Temp/processing_AjGEna/46f1679c887641ba8e8f177bfacff9df/OUTPUT.tif You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.

these were the error with clipping by layer by extent

njc888 commented 6 months ago

the next i tried was was clipping by clip raster by mask layer Input parameters: { 'ALPHA_BAND' : False, 'CROP_TO_CUTLINE' : True, 'DATA_TYPE' : 0, 'EXTRA' : '', 'INPUT' : 'C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Dem/N005E075_N010E080/N005E075_N010E080/ALPSMLC30_N008E077_DSM.tif', 'KEEP_RESOLUTION' : False, 'MASK' : 'C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/BoundingConvexTvmBorderDist.shp', 'MULTITHREADING' : False, 'NODATA' : None, 'OPTIONS' : '', 'OUTPUT' : 'TEMPORARY_OUTPUT', 'SET_RESOLUTION' : False, 'SOURCE_CRS' : None, 'TARGET_CRS' : None, 'TARGET_EXTENT' : None, 'X_RESOLUTION' : None, 'Y_RESOLUTION' : None }

GDAL command: gdalwarp -overwrite -of GTiff -cutline "C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/BoundingConvexTvmBorderDist.shp" -cl BoundingConvexTvmBorderDist -crop_to_cutline "C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Dem/N005E075_N010E080/N005E075_N010E080/ALPSMLC30_N008E077_DSM.tif" C:/Users/Power/AppData/Local/Temp/processing_AjGEna/865280054ec44d409716800365f08ab9/OUTPUT.tif GDAL command output: Creating output file that is 4678P x 4832L. Processing C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Dem/N005E075_N010E080/N005E075_N010E080/ALPSMLC30_N008E077_DSM.tif [1/1] : 0...10...20...30...40...50...60...70...80...90...100 - done. Process completed successfully Execution completed in 0.61 seconds Results: {'OUTPUT': 'C:/Users/Power/AppData/Local/Temp/processing_AjGEna/865280054ec44d409716800365f08ab9/OUTPUT.tif'}

Loading resulting layers Algorithm 'Clip raster by mask layer' finished

Presently it finished successfully however in the first attempt it ran for nearly two hr and my ssd nearly failed with smart warnings its output was in sdat format next time i tried it again went for two hrs without a succeful output. each time i cancelled, the QGIS crashed with the log report i last reported

njc888 commented 6 months ago

Though the output was shown as successful it was not correctly done

njc888 commented 6 months ago

GDAL command: gdalwarp -overwrite -t_srs EPSG:32643 -of GTiff -tr 0.00027777777777780147 -0.0002777777777777999 -tap -cutline C:/Users/Power/AppData/Local/Temp/processing_AjGEna/560f88f023a54f1782057a08a7013dce/MASK.gpkg -cl MASK -crop_to_cutline -multi "C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/Filled BorderDem.sdat" C:/Users/Power/AppData/Local/Temp/processing_AjGEna/c2b2623e04354ef2812dd155ad786e4a/OUTPUT.tif GDAL command output: ERROR 3: OUTPUT.tif: Free disk space available is 789532213248 bytes, whereas 480459922555843264 are at least necessary. You can disable this check by defining the CHECK_DISK_FREE_SPACE configuration option to FALSE. Creating output file that is 233036837P x 515433449L. Process returned error code 1 Results: { OUTPUT: 'C:/Users/Power/AppData/Local/Temp/processing_AjGEna/c2b2623e04354ef2812dd155ad786e4a/OUTPUT.tif' } OK. Execution took 5.386 s (1 output(s)). Model processed OK. Executed 5 algorithm(s) total in 5.667 s. Execution completed in 5.69 seconds Results: {'dem_mob': 'C:/Users/Power/AppData/Local/Temp/processing_AjGEna/c2b2623e04354ef2812dd155ad786e4a/OUTPUT.tif', 'mob_en': 'Clipped_8d63eae2_e118_42fe_88b7_a902ebf1fc81'}

Loading resulting layers The following layers were not correctly generated. • C:/Users/Power/AppData/Local/Temp/processing_AjGEna/c2b2623e04354ef2812dd155ad786e4a/OUTPUT.tif • Clipped_8d63eae2_e118_42fe_88b7_a902ebf1fc81 You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.

This error occurs again when i try to use the multithreaded implmentation of gdal in advanced options with keep resolution to match input raster as true

when i uncheck this othere errors crop up

njc888 commented 6 months ago

QGIS version: 3.34.3-Prizren QGIS code revision: 47373234ac Qt version: 5.15.3 Python version: 3.9.18 GDAL version: 3.8.3 GEOS version: 3.12.1-CAPI-1.18.1 PROJ version: Rel. 9.3.1, December 1st, 2023 PDAL version: 2.6.0 (git-version: 3fced5) Algorithm started at: 2024-03-22T14:16:46 Algorithm 'Clip raster by mask layer' starting… Input parameters: { 'ALPHA_BAND' : False, 'CROP_TO_CUTLINE' : True, 'DATA_TYPE' : 0, 'EXTRA' : '', 'INPUT' : 'C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/Filled BorderDem.sdat', 'KEEP_RESOLUTION' : True, 'MASK' : 'C:\Users\Power\Documents\MOB Siting\Kerala_hazard\Final_useful layers\New Data\TvmBordNoholes.shp', 'MULTITHREADING' : True, 'NODATA' : None, 'OPTIONS' : 'COMPRESS=DEFLATE|PREDICTOR=2|ZLEVEL=9', 'OUTPUT' : 'TEMPORARY_OUTPUT', 'SET_RESOLUTION' : False, 'SOURCE_CRS' : None, 'TARGET_CRS' : QgsCoordinateReferenceSystem('EPSG:32643'), 'TARGET_EXTENT' : None, 'X_RESOLUTION' : None, 'Y_RESOLUTION' : None }

GDAL command: gdalwarp -overwrite -t_srs EPSG:32643 -of GTiff -tr 0.00027777777777780147 -0.0002777777777777999 -tap -cutline "C:\Users\Power\Documents\MOB Siting\Kerala_hazard\Final_useful layers\New Data\TvmBordNoholes.shp" -cl TvmBordNoholes -crop_to_cutline -multi -co COMPRESS=DEFLATE -co PREDICTOR=2 -co ZLEVEL=9 "C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/Filled BorderDem.sdat" C:/Users/Power/AppData/Local/Temp/processing_RgMEQP/9def7050adcc4c75acf782c5a9278052/OUTPUT.tif GDAL command output: Process was canceled and did not complete Execution failed after 1414.03 seconds (23 minutes 34 seconds)

Loading resulting layers Algorithm 'Clip raster by mask layer' finished

njc888 commented 6 months ago

The algo rithm finishes in time when there is no change in the projection QGIS version: 3.34.3-Prizren QGIS code revision: 47373234ac Qt version: 5.15.3 Python version: 3.9.18 GDAL version: 3.8.3 GEOS version: 3.12.1-CAPI-1.18.1 PROJ version: Rel. 9.3.1, December 1st, 2023 PDAL version: 2.6.0 (git-version: 3fced5) Algorithm started at: 2024-03-22T14:44:13 Algorithm 'Clip raster by mask layer' starting… Input parameters: { 'ALPHA_BAND' : False, 'CROP_TO_CUTLINE' : True, 'DATA_TYPE' : 0, 'EXTRA' : '', 'INPUT' : 'C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/Filled BorderDem.sdat', 'KEEP_RESOLUTION' : True, 'MASK' : 'C:\Users\Power\Documents\MOB Siting\Kerala_hazard\Final_useful layers\New Data\TvmBordNoholes.shp', 'MULTITHREADING' : False, 'NODATA' : None, 'OPTIONS' : '', 'OUTPUT' : 'TEMPORARY_OUTPUT', 'SET_RESOLUTION' : False, 'SOURCE_CRS' : None, 'TARGET_CRS' : None, 'TARGET_EXTENT' : None, 'X_RESOLUTION' : None, 'Y_RESOLUTION' : None }

GDAL command: gdalwarp -overwrite -of GTiff -tr 0.00027777777777780147 -0.0002777777777777999 -tap -cutline "C:\Users\Power\Documents\MOB Siting\Kerala_hazard\Final_useful layers\New Data\TvmBordNoholes.shp" -cl TvmBordNoholes -crop_to_cutline "C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/Filled BorderDem.sdat" C:/Users/Power/AppData/Local/Temp/processing_RgMEQP/b384d6d81aa84e7bb2bf319f344dafba/OUTPUT.tif GDAL command output: Creating output file that is 4680P x 4834L. Processing C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/Filled BorderDem.sdat [1/1] : 0Using internal nodata values (e.g. -99999) for image C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/Filled BorderDem.sdat. Copying nodata values from source C:/Users/Power/Documents/MOB Siting/Kerala_hazard/Final_useful layers/New Data/Tn Distrcts Tvm Border/Filled BorderDem.sdat to destination C:/Users/Power/AppData/Local/Temp/processing_RgMEQP/b384d6d81aa84e7bb2bf319f344dafba/OUTPUT.tif. ...10...20...30...40...50...60...70...80...90...100 - done. Process completed successfully Execution completed in 1.34 seconds Results: {'OUTPUT': 'C:/Users/Power/AppData/Local/Temp/processing_RgMEQP/b384d6d81aa84e7bb2bf319f344dafba/OUTPUT.tif'}

Loading resulting layers Algorithm 'Clip raster by mask layer' finished