UMEP-dev / UMEP-processing

GNU General Public License v3.0
8 stars 9 forks source link

Urock stuck in 3D wind calculation for vegetation #93

Open biglimp opened 4 days ago

biglimp commented 4 days ago

We are running Urock in a loop in order to estimate wind speed for a whole city (Malmö, Sweden). Some grids is done very fast but then it gets stuck in one grid. Files attached. What I can see is that a .db file gets bigger and bigger until it has eaten up my entire harddrive which creates an memory error in a Java process.

@j3r3m1 , any idea what is going on? To me it seems like the process is in a infinite loop.

veg_urock73.zip

Here is the settings used:

parin = { 'ATTENUATION_FIELD' : '', 
             'BUILDINGS' : buildname, 
             'HEIGHT_FIELD_BUILD' : 'ROOF_H', 
             'HORIZONTAL_RESOLUTION' : 4, 
             'INPUT_PROFILE_FILE' : '', 
             'INPUT_PROFILE_TYPE' : 0, 
             'INPUT_WIND_DIRECTION' : 230, 
             'INPUT_WIND_HEIGHT' : 10, 
             'INPUT_WIND_SPEED' : 2, 
             'LOAD_OUTPUT' : False, 
             'OUTPUT_FILENAME' : 'ws_1_5magl_' + str(gridid), 
             'RASTER_OUTPUT' : filepath_tempdsm, 
             'SAVE_NETCDF' : False, 
             'SAVE_RASTER' : True, 
             'SAVE_VECTOR' : False, 
             'UROCK_OUTPUT' : output_dir_urock, 
             'VEGETATION' : vegname, 
             'VEGETATION_CROWN_BASE_HEIGHT' : '', 
             'VEGETATION_CROWN_TOP_HEIGHT' : 'VEG_H', 
             'VERTICAL_RESOLUTION' : 4, 
             'WIND_HEIGHT' : '1.5' }
    processing.run("umep:Urban Wind Field: URock", parin)

Here are the log print out:

[Connecting to database
        ->C:\Users\xlinfr\AppData\Local\Temp\myDbH21732886320_5743434
C:\Users\xlinfr\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\processing_umep\functions\URock\h2gis-standalone\h2gis-dist-2.2.3.jar
Connected!

Spatial functions added!

Load input data
Load table 'build_pre_srid_20241129141840'
Load table 'veg_pre_srid_20241129141840'
Creates blocks and stacked blocks
Rotates geometries from 230.0 degrees
Identify block base height and block cavity base
Calculates obstacle properties
Calculates zone properties
Initializes upwind facades
Update upwind facades base height
Initializes downwind facades
Calculates study area properties
Rotates geometries from -230.0 degrees
Creates displacement zones
Creates cavity and wake zones
Creates street canyon zones
Creates rooftop zones (perpendicular and corner)
Creates built-up and open vegetation zones
Identify the buildings concerned by the impacted zone chosen by the user
Creates the grid of points
Affects each grid point to a building Rockle zone and calculates needed 
          variables for 3D wind speed
Affects each grid point to a vegetation Rockle zone and calculates 
          needed variables for 3D wind speed
Remove some of the Röckle zone points
Creates backward zones
Calculates the 3D wind speed factor value for each point of each BUILDING zone
Calculates the 3D wind speed factor value for each point of each VEGETATION zone](url)

Here is a detailed error message from VSCode when memory is full:

    MainCalculation.main(javaEnvironmentPath = javaEnvVar,
  File "C:\Users\xlinfr\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\processing_umep\functions\URock\MainCalculation.py", line 540, in main
    InitWindField.calculates3dVegWindFactor(cursor = cursor,
  File "C:\Users\xlinfr\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\processing_umep\functions\URock\InitWindField.py", line 1892, in calculates3dVegWindFactor     
    cursor.execute(";".join(["""
  File "C:\OSGeo4W\apps\Python312\Lib\site-packages\jaydebeapi\__init__.py", line 536, in execute
    _handle_sql_exception()
  File "C:\OSGeo4W\apps\Python312\Lib\site-packages\jaydebeapi\__init__.py", line 165, in _handle_sql_exception_jpype
    reraise(exc_type, exc_info[1], exc_info[2])
  File "C:\OSGeo4W\apps\Python312\Lib\site-packages\jaydebeapi\__init__.py", line 57, in reraise
    raise value.with_traceback(tb)
  File "C:\OSGeo4W\apps\Python312\Lib\site-packages\jaydebeapi\__init__.py", line 534, in execute
    is_rs = self._prep.execute()
            ^^^^^^^^^^^^^^^^^^^^
jaydebeapi.DatabaseError: org.h2.jdbc.JdbcSQLNonTransientException: IO Exception: "C:/Users/xlinfr/AppData/Local/Temp/myDbH21732873896_8944085.mv.db"; SQL statement:
CREATE TABLE VEGETATION_OPEN_TEMPO_3DPOINTS
               AS SELECT b.ID_POINT, a.ID_Z,  CASE WHEN   a.Z>b.MAX_HEIGHT
                    THEN    LOG((a.Z - 3 * 0.05 * MAX_CANOPY_HEIGHT) / 0.9839647414215449) / LOG(a.Z / 0.9839647414215449)
                    ELSE    CASE WHEN   a.Z > b.MAX_HEIGHT OR a.Z < b.MIN_HEIGHT
                            THEN        LOG((b.MAX_CANOPY_HEIGHT - 3 * 0.05 * MAX_CANOPY_HEIGHT) / 0.9839647414215449) / LOG(a.Z / 0.9839647414215449)
                            ELSE        LOG((b.MAX_HEIGHT - 3 * 0.05 * MAX_CANOPY_HEIGHT) / 0.9839647414215449) / LOG(a.Z / 0.9839647414215449) * EXP(b.ATTENUATIO * (a.Z / b.MAX_CANOPY_HEIGHT - 1))
                    END
                END
             AS VEGETATION_FACTOR
               FROM Z_VALUES_20241129105157 AS a, VEGETATION_OPEN_POINTS AS b
               WHERE a.Z > 0 [90028-232]

Traceback (most recent call last):
  File "f:\Scripts\1 - SOLWEIG-automation\UROCK-Automation.py", line 209, in <module>
    processing.run("umep:Urban Wind Field: URock", parin)
  File "C:\OSGeo4W\apps\qgis\python\plugins\processing\tools\general.py", line 93, in run
    return Processing.runAlgorithm(algOrName, parameters, onFinish, feedback, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\qgis\python\plugins\processing\core\Processing.py", line 192, in runAlgorithm
    ret, results = execute(alg, parameters, context, feedback, catch_exceptions=False)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\qgis\python\plugins\processing\gui\AlgorithmExecutor.py", line 70, in execute
    results, ok = alg.run(parameters, context, feedback, {}, False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_core.QgsProcessingException: Traceback (most recent call last):
  File "Thread.java", line 1583, in java.lang.Thread.run
java.io.java.io.IOException: java.io.IOException: There is not enough space on the disk

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "Thread.java", line 1583, in java.lang.Thread.run
org.h2.mvstore.org.h2.mvstore.MVStoreException: org.h2.mvstore.MVStoreException: Writing to sun.nio.ch.FileChannelImpl@41ff4f2c failed; length 2826240 at 12238094336 [2.3.232/2] 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "JdbcPreparedStatement.java", line 262, in org.h2.jdbc.JdbcPreparedStatement.execute
Exception: Java Exception

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\OSGeo4W\apps\Python312\Lib\site-packages\jaydebeapi\__init__.py", line 534, in execute
    is_rs = self._prep.execute()
            ^^^^^^^^^^^^^^^^^^^^
org.h2.jdbc.org.h2.jdbc.JdbcSQLNonTransientException: org.h2.jdbc.JdbcSQLNonTransientException: IO Exception: "C:/Users/xlinfr/AppData/Local/Temp/myDbH21732873896_8944085.mv.db"; SQL statement:
CREATE TABLE VEGETATION_OPEN_TEMPO_3DPOINTS
               AS SELECT b.ID_POINT, a.ID_Z,  CASE WHEN   a.Z>b.MAX_HEIGHT
                    THEN    LOG((a.Z - 3 * 0.05 * MAX_CANOPY_HEIGHT) / 0.9839647414215449) / LOG(a.Z / 0.9839647414215449)
                    ELSE    CASE WHEN   a.Z > b.MAX_HEIGHT OR a.Z < b.MIN_HEIGHT
                            THEN        LOG((b.MAX_CANOPY_HEIGHT - 3 * 0.05 * MAX_CANOPY_HEIGHT) / 0.9839647414215449) / LOG(a.Z / 0.9839647414215449)
                            ELSE        LOG((b.MAX_HEIGHT - 3 * 0.05 * MAX_CANOPY_HEIGHT) / 0.9839647414215449) / LOG(a.Z / 0.9839647414215449) * EXP(b.ATTENUATIO * (a.Z / b.MAX_CANOPY_HEIGHT - 1))
                    END
                END
             AS VEGETATION_FACTOR
               FROM Z_VALUES_20241129105157 AS a, VEGETATION_OPEN_POINTS AS b
               WHERE a.Z > 0 [90028-232]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\xlinfr\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\processing_umep\processor\urock_processing_algorithm.py", line 447, in processAlgorithm        
    MainCalculation.main(javaEnvironmentPath = javaEnvVar,
  File "C:\Users\xlinfr\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\processing_umep\functions\URock\MainCalculation.py", line 540, in main
    InitWindField.calculates3dVegWindFactor(cursor = cursor,
  File "C:\Users\xlinfr\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\processing_umep\functions\URock\InitWindField.py", line 1892, in calculates3dVegWindFactor     
    cursor.execute(";".join(["""
  File "C:\OSGeo4W\apps\Python312\Lib\site-packages\jaydebeapi\__init__.py", line 536, in execute
    _handle_sql_exception()
  File "C:\OSGeo4W\apps\Python312\Lib\site-packages\jaydebeapi\__init__.py", line 165, in _handle_sql_exception_jpype
    reraise(exc_type, exc_info[1], exc_info[2])
  File "C:\OSGeo4W\apps\Python312\Lib\site-packages\jaydebeapi\__init__.py", line 57, in reraise
    raise value.with_traceback(tb)
  File "C:\OSGeo4W\apps\Python312\Lib\site-packages\jaydebeapi\__init__.py", line 534, in execute
    is_rs = self._prep.execute()
            ^^^^^^^^^^^^^^^^^^^^
jaydebeapi.DatabaseError: org.h2.jdbc.JdbcSQLNonTransientException: IO Exception: "C:/Users/xlinfr/AppData/Local/Temp/myDbH21732873896_8944085.mv.db"; SQL statement:
CREATE TABLE VEGETATION_OPEN_TEMPO_3DPOINTS
               AS SELECT b.ID_POINT, a.ID_Z,  CASE WHEN   a.Z>b.MAX_HEIGHT
                    THEN    LOG((a.Z - 3 * 0.05 * MAX_CANOPY_HEIGHT) / 0.9839647414215449) / LOG(a.Z / 0.9839647414215449)
                    ELSE    CASE WHEN   a.Z > b.MAX_HEIGHT OR a.Z < b.MIN_HEIGHT
                            THEN        LOG((b.MAX_CANOPY_HEIGHT - 3 * 0.05 * MAX_CANOPY_HEIGHT) / 0.9839647414215449) / LOG(a.Z / 0.9839647414215449)
                            ELSE        LOG((b.MAX_HEIGHT - 3 * 0.05 * MAX_CANOPY_HEIGHT) / 0.9839647414215449) / LOG(a.Z / 0.9839647414215449) * EXP(b.ATTENUATIO * (a.Z / b.MAX_CANOPY_HEIGHT - 1))
                    END
                END
             AS VEGETATION_FACTOR
               FROM Z_VALUES_20241129105157 AS a, VEGETATION_OPEN_POINTS AS b
               WHERE a.Z > 0 [90028-232]
biglimp commented 1 day ago

Just to update. A new run was done and now it came further but stopped, at a grid when the db-file ate up the harddrive memory (myDbH21732916050_978621.mv.db = 75Gb). What was interesting with this run was that it came through one problem-grid where it stopped before (maybe because I removed files and increased my harddrive space).

j3r3m1 commented 1 day ago

There was two issues:

Thus it results in a domain too big along the vertical axis. I have now found the bug but I have no time this week to resolve. An option could be to remove these zones for now (I do not think it affects much the results on the ground).

biglimp commented 1 day ago

Great that you found the bugs. No hurry. Let us know when you fixed so that we can try again.

j3r3m1 commented 1 day ago

If needed, here is the code without rooftop corner: https://github.com/UMEP-dev/UMEP-processing/tree/URock_noroofcorner