search

UMEP-dev / UMEP

Urban Multi-scale Environmental Predictor
https://umep-docs.readthedocs.io/
62 stars 15 forks source link

Unable to allocate 5.66GiB space #371

Closed phemiobe closed 2 years ago

phemiobe commented 2 years ago

Hello UMEP community, When running the SUEWS model with 9842 grids at 1km resolution; I have this error. 

Unable to allocate 5.66 GiB for an array with shape (440, 1726272) and data type float64

Check: C:\Users\User\Documents\SuPy.log

Please report any errors to https://github.com/UMEP-dev/UMEP/issues

Does this meant that the capacity of my machine is insufficient?

sunt05 commented 2 years ago

Hi @phemiobe , you guess is right: this looks to me to be a memory issue. Please try to reduced the number of simulation grids.

Also, please report your supy version info by running supy.show_version() in the QGIS python console so I can better help you if further issue appears.

phemiobe commented 2 years ago

Number of grids have been reduced yet I have the same error.. here I have 966 grid points at 2km resolution

Unable to allocate 7.30 GiB for an array with shape (440, 2225664) and data type float64

Check: C:\Users\User\Documents\SuPy.log Pasted below is my supy version

SuPy versions

supy: 2022.2.17 supy_driver: 2021a2

================= SYSTEM DEPENDENCY

INSTALLED VERSIONS

commit : 06d230151e6f18fdb8139d09abf539867a8cd481 python : 3.9.5.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19044 machine : AMD64 processor : Intel64 Family 6 Model 78 Stepping 3, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : English_Ireland.1252

pandas : 1.4.1 numpy : 1.21.5 pytz : 2020.1 dateutil : 2.8.1 pip : 21.3.1 setuptools : 56.0.0 Cython : 0.29.23 pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : 4.6.2 html5lib : None pymysql : None psycopg2 : 2.8.6 jinja2 : 3.0.3 IPython : 8.1.1 pandas_datareader: None bs4 : None bottleneck : None fastparquet : None fsspec : 2022.01.0 gcsfs : None matplotlib : 3.5.1 numba : 0.55.1 numexpr : 2.8.1 odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pyreadstat : None pyxlsb : None s3fs : None scipy : 1.6.2 sqlalchemy : 1.4.31 tables : None tabulate : 0.8.9 xarray : 0.20.2 xlrd : 1.2.0 xlwt : 1.3.0 zstandard : None

sunt05 commented 2 years ago

@phemiobe can you please attach your supy.log file here for me to see some error details? thanks!

phemiobe commented 2 years ago

Thanks, please find attached file SuPy.log

phemiobe commented 2 years ago

here is my machine specification

Device name DESKTOP-K9T76PC Processor Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz 2.40 GHz Installed RAM 8.00 GB Pen and touch No pen or touch input is available for this display

sunt05 commented 2 years ago

and a full error message stack from python please.

sunt05 commented 2 years ago

are you using UMEP for the simulation or standalone supy?

phemiobe commented 2 years ago

am using UMEP and I dont have python error. I only got that message after about 40 minutes of run

sunt05 commented 2 years ago

Just for testing-if you try only one grid, would the same issue occur?

I'm wondering where the issue could be: number of grids or length of simulation period.

phemiobe commented 2 years ago

when I tried reduced the grid drastically to like 330 at 5km, it worked. this is however to coarse for me. any advice pls

phemiobe commented 2 years ago

Please how can I run standalone supy?

sunt05 commented 2 years ago

when I tried reduced the grid drastically to like 330 at 5km, it worked. this is however to coarse for me. any advice pls

Good to know this; and the solution is obvious then: you just need to split your grids-but keep them in your desired resolution-into multiple batches and run them batch by batch.

sunt05 commented 2 years ago

Please how can I run standalone supy?

Please see the SuPy docs site: https://supy.readthedocs.io/en/latest/

phemiobe commented 2 years ago

Hello, I tried with a bigger machine(64GB ram). the issue still persists. did any else experience this..... SuPy.log experience this allocate

sunt05 commented 2 years ago

Sorry to see this.

If you can run supy outside of QIGS with the same setting on this machine, then I gather there might be something to do with the memory management of QGIS, which, unfortunately, is totally beyond our control.

However, if you see a similar error with the standalone supy, please report here with your supy version (supy.show_version()) and your input files: I'm happy to have a look for the cause.

phemiobe commented 2 years ago

Hello Ting

I followed the supy documentation and had my first successful run with standalone supy. However, I suppose the output would be saved in the FileOutput Path of the RunControl,nml. the folder was empty after the simulation but I was able to visualize the dfoutput. Please how do I locate the output folder since its not saved in the FileOutputPath pasted below.

Thanks

FileOutputPath="/home/people/Documents/SUEWS_Project/SUEWS_2020/output"

sunt05 commented 2 years ago

Hi there,

in SuPy, running simulations and saving results are separate processes and thus are dealt with by different functions: please see save_supy for more details.

sunt05 commented 2 years ago

Hi @phemiobe, if this issue has been resolved, please consider closing the issue. thanks!