Urock prepare: problems tmp/ folder?

[matthiasdemuzere]

Hi @j3r3m1, all,

I am currently trying the URock tutorial, using default settings and sample data.

When doing so, I get a rather vague error:

QGIS version: 3.34.3-Prizren
Qt version: 5.15.8
Python version: 3.11.7
GDAL version: 3.8.3
GEOS version: 3.12.1-CAPI-1.18.1
PROJ version: Rel. 9.3.1, December 1st, 2023
PDAL version: 2.6.2 (git-version: 3bfb98)

Algorithm started at: 2024-04-12T12:52:01
Algorithm 'Urban Wind Field: URock Prepare' starting…

Input parameters:
{ 'BUILDINGS_WITH_HEIGHT' : '/tmp/build_vector.GeoJSON', 'HEIGHT_VEG_FIELD' : '', 'INPUT_BUILD_DEM' : '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/dem.tif', 'INPUT_BUILD_DSM' : '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/dsm.tif', 'INPUT_BUILD_FOOTPRINT' : '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/buildings.shp', 'INPUT_VEG_CDSM' : '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/cdsm.tif', 'INPUT_VEG_POINTS' : None, 'OUTPUT_BUILD_HEIGHT_FIELD' : 'ROOF_HEIGHT', 'OUTPUT_VEG_HEIGHT_FIELD' : 'VEG_HEIGHT', 'RADIUS_VEG_FIELD' : '', 'VEGETATION_ASPECT' : '0.75', 'VEGETATION_WITH_HEIGHT' : '/tmp/veg_vector.GeoJSON' }

Unable to execute algorithm
Could not load source layer for INPUT_RASTER: /tmp/processing_RGpzrq/c4ca8de0363b4cf18352e2fc21654962/OUTPUT.ter not found

Execution failed after 0.28 seconds

I checked the urock_prepare_algorihm.py a bit, and the issue seems to arise at build_dsm = processing.run("gdal:rastercalculator", ...

First I test the raster calculation directly from gdal, used within the same environment as qgis: gdal_calc.py -A dsm.tif -B dem.tif --outfile=dsm-dem.tif --calc="(A > B) * (A - B) + (A <= B) * 0" That works.

After some testing I found out that writing to tmp/ dir is problematic? The file is created on my system, but it is empty. When changing 'OUTPUT':'TEMPORARY_OUTPUT' to an actual directory, eg. 'OUTPUT':'SOME/DIR/tmp.tif'

That seems to work. Although then another error dissapears (at least when using the python console):

_core.QgsProcessingException: Could not create raster output /tmp/processing_RGpzrq/81848f71c6a24c27a34218a3a7063b94/OUTPUT.ter: GDAL provider Cannot create new dataset  /tmp/processing_RGpzrq/81848f71c6a24c27a34218a3a7063b94/OUTPUT.ter:
MINUSERPIXELVALUE must be specified

Though I don't think this one is relevant at the moment.

In any case, when setting a non-tmp directory to when creating build_dsm, and running the GUI approach again, build_vector.GeoJSON is correctly created. veg_vector.GeoJSON not yet:

QGIS version: 3.34.3-Prizren
Qt version: 5.15.8
Python version: 3.11.7
GDAL version: 3.8.3
GEOS version: 3.12.1-CAPI-1.18.1
PROJ version: Rel. 9.3.1, December 1st, 2023
PDAL version: 2.6.2 (git-version: 3bfb98)
Algorithm started at: 2024-04-12T16:48:01
Algorithm 'Urban Wind Field: URock Prepare' starting…
Input parameters:
{ 'BUILDINGS_WITH_HEIGHT' : '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/tmp/build_vector.GeoJSON', 'HEIGHT_VEG_FIELD' : '', 'INPUT_BUILD_DEM' : '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/dem.tif', 'INPUT_BUILD_DSM' : '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/dsm.tif', 'INPUT_BUILD_FOOTPRINT' : '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/buildings.shp', 'INPUT_VEG_CDSM' : None, 'INPUT_VEG_POINTS' : None, 'OUTPUT_BUILD_HEIGHT_FIELD' : 'ROOF_HEIGHT', 'OUTPUT_VEG_HEIGHT_FIELD' : 'VEG_HEIGHT', 'RADIUS_VEG_FIELD' : '', 'VEGETATION_ASPECT' : '0.75', 'VEGETATION_WITH_HEIGHT' : '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/tmp/veg_vector.GeoJSON' }

Execution completed in 0.29 seconds
Results:
{'BUILDINGS_WITH_HEIGHT': '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/tmp/build_vector.GeoJSON',
'VEGETATION_WITH_HEIGHT': '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/tmp/veg_vector.GeoJSON'}

Loading resulting layers
The following layers were not correctly generated.
• /mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/tmp/veg_vector.GeoJSON
You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.

Also setting a proper path in:

                veg_vect = processing.run("native:pixelstopolygons", 
                                          {'INPUT_RASTER':veg_dsm_rounded,
                                           'RASTER_BAND':1,
                                           'FIELD_NAME':'VALUE',
#                                           'OUTPUT':'TEMPORARY_OUTPUT'})["OUTPUT"]
                                           'OUTPUT':'/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/tmp/tmp_vect.tif'})["OUTPUT"]

Seems to work for me.

So ... long story short: seems some of the processes are struggling to create the files in the /tmp/ directory? At least on my ubuntu 22.04 system?

I am not sure why that is, since this directory is accessible and writeable ... Any ideas?

[matthiasdemuzere]

Perhaps just to clarify: it is not a problem with the /tmp/*_vector.GeoJSON files. When I use the default, they are correctly written to the tmp/ folder.

It seems to be a problem with the temporary files created within the Algorithm ...

[j3r3m1]

ing a proper path in:

                veg_vect = processing.run("native:pixelstopolygons", 
                                          {'INPUT_RASTER':veg_dsm_rounded,
                                           'RASTER_BAND':1,
                                           'FIELD_NAME':'VALUE',
#                                           'OUTPUT':'TEMPORARY_OUTPUT'})["OUTPUT"]
                                           'OUTPUT':'/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/tmp/tmp_vect.tif'})["OUTPUT"]

Hi @matthiasdemuzere thanks for reporting and investigating.

Is is the same error if you replace 'TEMPORARY_OUTPUT' by a file in the temporary output (such as '/tmp/test.tif') or is it the same problem (does it come from the directory automatically created by QGIS or by saving anything in the temp folder) ?

[matthiasdemuzere]

It works when defining my own temp file, eg. /tmp/tmp.tif. But not when using the TEMPORARY_OUTPUT.

As far as I can see, this behaviour occurs in the following algorithms:

• "gdal:rastercalculator"
• "native:roundrastervalues"

This is confirmed when launching these separately in QGIS, eg. for rastercalculator:

Saving as temporary file:

Algorithm started at: 2024-04-15T09:51:09
Algorithm 'Raster calculator' starting…
Input parameters:
{ 'BAND_A' : 1, 'BAND_B' : None, 'BAND_C' : None, 'BAND_D' : None, 'BAND_E' : None, 'BAND_F' : None, 'EXTENT_OPT' : 0, 'EXTRA' : '', 'FORMULA' : '(A > B) * (A - B) + (A <= B) * 0', 'INPUT_A' : '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/dsm.tif', 'INPUT_B' : '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/dem.tif', 'INPUT_C' : None, 'INPUT_D' : None, 'INPUT_E' : None, 'INPUT_F' : None, 'NO_DATA' : None, 'OPTIONS' : '', 'OUTPUT' : 'TEMPORARY_OUTPUT', 'PROJWIN' : None, 'RTYPE' : 5 }

GDAL command:
gdal_calc.py --overwrite --calc "(A > B) * (A - B) + (A <= B) * 0" --format Leveller --type Float32 -A /mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/dsm.tif --A_band 1 -B /mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/dem.tif --outfile /tmp/processing_rolgHG/1c8e70e9d4e64635a5bab0185472df79/OUTPUT.ter
GDAL command output:
Traceback (most recent call last):
File "/home/matthias/micromamba/envs/qgis-env/lib/python3.11/site-packages/osgeo_utils/auxiliary/gdal_argparse.py", line 226, in main
self.doit(**kwargs)
File "/home/matthias/micromamba/envs/qgis-env/lib/python3.11/site-packages/osgeo_utils/gdal_calc.py", line 887, in doit
return Calc(**kwargs)
^^^^^^^^^^^^^^
File "/home/matthias/micromamba/envs/qgis-env/lib/python3.11/site-packages/osgeo_utils/gdal_calc.py", line 424, in Calc
myOut = myOutDrv.Create(
^^^^^^^^^^^^^^^^
File "/home/matthias/micromamba/envs/qgis-env/lib/python3.11/site-packages/osgeo/gdal.py", line 3192, in Create
return _gdal.Driver_Create(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: MINUSERPIXELVALUE must be specified.
Process returned error code 1
Execution completed in 0.32 seconds
Results:
{'OUTPUT': '/tmp/processing_rolgHG/1c8e70e9d4e64635a5bab0185472df79/OUTPUT.ter'}

Loading resulting layers
The following layers were not correctly generated.
• /tmp/processing_rolgHG/1c8e70e9d4e64635a5bab0185472df79/OUTPUT.ter
You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.

Defining an output file

Algorithm 'Raster calculator' starting…
Input parameters:
{ 'BAND_A' : 1, 'BAND_B' : None, 'BAND_C' : None, 'BAND_D' : None, 'BAND_E' : None, 'BAND_F' : None, 'EXTENT_OPT' : 0, 'EXTRA' : '', 'FORMULA' : '(A > B) * (A - B) + (A <= B) * 0', 'INPUT_A' : '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/dsm.tif', 'INPUT_B' : '/mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/dem.tif', 'INPUT_C' : None, 'INPUT_D' : None, 'INPUT_E' : None, 'INPUT_F' : None, 'NO_DATA' : None, 'OPTIONS' : '', 'OUTPUT' : '/home/matthias/.local/share/QGIS/QGIS3/profiles/default/processing/outputs/test.tif', 'PROJWIN' : None, 'RTYPE' : 5 }

GDAL command:
gdal_calc.py --overwrite --calc "(A > B) * (A - B) + (A <= B) * 0" --format GTiff --type Float32 -A /mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/dsm.tif --A_band 1 -B /mnt/SSD2TB/data/UMEP/UROCK/Annedal_EPSG3006/dem.tif --outfile /home/matthias/.local/share/QGIS/QGIS3/profiles/default/processing/outputs/test.tif
GDAL command output:
0...10...20...30...40...50...60...70...80...90...100 - done.
Process completed successfully
Execution completed in 0.38 seconds
Results:
{'OUTPUT': '/home/matthias/.local/share/QGIS/QGIS3/profiles/default/processing/outputs/test.tif'}

PS. Note on the side:

Is it intended to do roundrastervalues twice on the CDSM? As now done in line 331 and 338?

[biglimp]

@matthiasdemuzere , one thing that might work is to change default filetype for raster. In WGIS, go to Settings/Options/Processing/General and make sure that you are using .tif as Default output raster layer extension.

[matthiasdemuzere]

@biglimp Thanks, this seems work? At least when testing on the rastercalulcator directly. I'll set everything to default again in urock_prepare_algorithm.py and will try again.

[matthiasdemuzere]

@matthiasdemuzere , one thing that might work is to change default filetype for raster. In WGIS, go to Settings/Options/Processing/General and make sure that you are using .tif as Default output raster layer extension.

Great, this solution worked for me! Thanks again.

[biglimp]

@matthiasdemuzere , one thing that might work is to change default filetype for raster. In WGIS, go to Settings/Options/Processing/General and make sure that you are using .tif as Default output raster layer extension.

Great, this solution worked for me! Thanks again.

Great. @j3r3m1 , we should communicate this to other users, to use .tif and .shp as standard.

[j3r3m1]

Great. @j3r3m1 , we should communicate this to other users, to use .tif and .shp as standard.

It can be in the manual (for example a new 2.2 section called "Recommended default QGIS output files" - https://umep-docs.readthedocs.io/en/latest/Getting_Started.html#getting-started) and also in the note of some of the plugins that may cause the problem (e.g.: https://umep-docs.readthedocs.io/projects/tutorial/en/latest/Tutorials/IntroductionToURock.html) ?

[j3r3m1]

It can also be added in the FAQ...

[biglimp]

It can also be added in the FAQ...

Yes. I can add it in FAQ

[j3r3m1]

Do I add it also in the getting started and in the note of some of the models or FAQ is sufficient ?

[biglimp]

Do I add it also in the getting started and in the note of some of the models or FAQ is sufficient ?

Yes, maybe good to add in getting started also.

[j3r3m1]

Yes, maybe good to add in getting started also.

I do that right away