memory issue when running supy for large scale simulations

[sunt05]

I have 32 GB installed memory on my laptop.

Originally posted by @shahjahan0275 in https://github.com/UMEP-dev/UMEP/issues/462#issuecomment-1290282343

[sunt05]

similar upstream issue: https://github.com/UMEP-dev/SuPy/issues/47

[sunt05]

to look into this issue, please try out the approach here first:

https://github.com/UMEP-dev/SuPy/issues/47#issuecomment-1189572983

[biglimp]

@shahjahan0275 could you try to run via UMEP for processing instead? It is a plugin the make use of UMEP via the Processing toolbox.

[shahjahan0275]

I ran via UMEP for processing and it failed "Execution failed after 2711.37 seconds (45 minutes 11 seconds)"

[shahjahan0275]

I will now try the workaround, splitting the case into smaller chunks as suggested by @sunt05. I will reduce the weather file to 1 month

[shahjahan0275]

I have deleted 11 months of data from weather file and kept the data of January 1-February 1, 2010. The model ran. but in the last part of tutorial I got the following error 2022-10-26T12:27:13 WARNING Python error : An error has occurred while executing Python code: See message log (Python Error) for more details. 2022-10-26T12:27:13 WARNING Traceback (most recent call last): File "C:\Users/Shahjahan/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP-SuPy-QGIS3\SUEWSAnalyzer\suewsanalyzer.py", line 476, in spatial datawhole = np.genfromtxt(self.fileoutputpath + '/' + self.fileCode + gid + '' File "C:\OSGeo4W\apps\Python39\lib\site-packages\numpy\lib\npyio.py", line 1791, in genfromtxt fid = np.lib._datasource.open(fname, 'rt', encoding=encoding) File "C:\OSGeo4W\apps\Python39\lib\site-packages\numpy\lib_datasource.py", line 194, in open return ds.open(path, mode, encoding=encoding, newline=newline) File "C:\OSGeo4W\apps\Python39\lib\site-packages\numpy\lib_datasource.py", line 531, in open raise IOError("%s not found." % path) OSError: E:\BUET\Arch 6105\SUEWSWUDAPT_NYC\SUEWS_Output//nyc1_2010_SUEWS_60.txt not found. Attaching my reduced weather file also nyc_2010_data_60.txt

[shahjahan0275]

** small correction, in the weather file I kept the data from January 1 to January 31. Regards

[shahjahan0275]

in the processing tab of QGIS I also found following error 2022-10-26T12:22:06 WARNING Duplicate parameter INPUT_DEM registered for alg Outdoor Thermal Comfort: SOLWEIG 2022-10-26T12:22:20 CRITICAL Problem with OTB installation: OTB folder is not set.

[biglimp]

The INPUT_DEM is from another model which is not affected by you. THe OBT come from Orfeo Toolbox being not set which is also not affecting this issue.

The SUEWS Analyzer issue. Can you show me the content of your output folder after running the model

Also, could you please try something else for us: We suspect that it is the active QGIS-session that is causing the memory issue. With your input data (full year) that you have produced via the WUDAPT-tutorial, try to execute SUEWS from outside of QGIS. Here is how you do it:

1. Locate RunControl.nml that is used for your model run. I should be located in C:\Users**your user name**\AppData\Roaming\QGIS\QGIS3\profiles\default\python\plugins\UMEP\suewsmodel\Input
2. Open OSGeo4WShell which you can find from the start menu in Windows
3. Type python
4. Then you should follow the workflow from https://supy.readthedocs.io/ by typing:

   import supy as sp
   from pathlib import Path
   path_runcontrol = Path('C:/Users/**your user name**/AppData/Roaming/QGIS/QGIS3/profiles/default/python/plugins/UMEP/suewsmodel') / 'RunControl.nml'
   df_state_init = sp.init_supy(path_runcontrol)
   grid=df_state_init.index[0]
   df_forcing = sp.load_forcing_grid(path_runcontrol, grid)
   df_output,df_state_final=sp.run_supy(df_forcing, df_state_init)

[shahjahan0275]

Below is both input and output file. Also I will run the model as you advised. SUEWS_Output.zip SUEWS_Input.zip

[shahjahan0275]

is it the correct way of running model in supy:

[shahjahan0275]

got this after running for couple of hours:

[sunt05]

In this case, it might be more helpful to split your df_state into multiple single-grid dataframes and run them sequentially - for each single grid run, save the results as external files and load them back in for further analysis.

[biglimp]

Thank you @shahjahan0275 , now we know that it is a windows issue rather than a QGIS issue. Will will work on a solution. In the mean time make use of the solution suggested by @sunt05 of splitting your data.

I will try to sort out your issue in the SUEWS Analyzer as soon as possible.

[biglimp]

The reason SUEWSAnalyzer gives error is because you have created your grid in some strange way so that is is not covering the LCZ-data. This means that you dont have data for grid 1,7 etc. see attached screen dump and SiteSelect.txt used in SUEWS.

You need to make a new grid that is completely within the LCZdata. If the text in the tutorial is wrong, please submit an issue to the documentation and we will clarify the tutorial text.