SUPY ERROR"tables.exceptions.HDF5ExtError: Problems creating the Array"

[phemiobe]

Hello, I am running supy and encountered HDF5EXTError, could you please advise, perhaps you have encountered this type of error before. the full python error is attached as well as the supy version. Thanks

>>> df_output, df_state_final = sp.run_supy(df_forcing, df_state_init)
2022-03-22 14:37:37,404 - SuPy - INFO - ====================
2022-03-22 14:37:37,404 - SuPy - INFO - Simulation period:
2022-03-22 14:37:37,405 - SuPy - INFO -   Start: 2000-02-19 23:05:00
2022-03-22 14:37:37,405 - SuPy - INFO -   End: 2000-02-27 23:00:00
2022-03-22 14:37:37,405 - SuPy - INFO -
2022-03-22 14:37:37,405 - SuPy - INFO - No. of grids: 108535
2022-03-22 14:37:37,406 - SuPy - INFO - SuPy is running in parallel mode

SuPy versions
-------------
supy: 2022.3.17
supy_driver: 2021a4

=================
SYSTEM DEPENDENCY
/opt/software/python/3.7.4/lib/python3.7/site-packages/distributed/config.py:20: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  defaults = yaml.load(f)

INSTALLED VERSIONS
------------------
commit           : 66e3805b8cabe977f40c05259cc3fcf7ead5687d
python           : 3.7.4.final.0
python-bits      : 64
OS               : Linux
OS-release       : 3.10.0-1160.15.2.el7.x86_64
Version          : #1 SMP Thu Jan 21 16:15:07 EST 2021
machine          : x86_64
processor        :
byteorder        : little
LC_ALL           : None
LANG             : en_GB.UTF-8
LOCALE           : en_GB.UTF-8

pandas           : 1.3.5
numpy            : 1.20.0
pytz             : 2019.1
dateutil         : 2.8.0
pip              : 21.1.3
setuptools       : 41.4.0
Cython           : 0.29.20
pytest           : 5.0.1
hypothesis       : None
sphinx           : 2.1.2
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10.1
IPython          : 7.6.1
pandas_datareader: None
bs4              : 4.9.3
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : 3.0.3
numexpr          : 2.7.1
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyxlsb           : None
s3fs             : None
scipy            : 1.4.1
sqlalchemy       : None
tables           : 3.6.1
tabulate         : 0.8.5
xarray           : 0.20.2
xlrd             : None
xlwt             : None
numba            : 0.52.0

ERROR.txt

[phemiobe]

solved when I reduced the number of grids

[phemiobe]

However I have another issue when saving ....df_output with list_path_save = supy.save_supy(df_output, df_state_final, freq_s=3600, site='', path_dir_save='path/to/some/dir'). I got numpy.core._exceptions.MemoryError: Unable to allocate 180. GiB for an array with shape (384, 62913024) and data type float64

I knew this error is related to memory therefore I selected just SUEWS df_output_suews = df_output['SUEWS']

and then list_path_save = sp.save_supy(df_output_suews, df_state_final, freq_s=3600, site='', path_dir_save='/home/people/20202815/scratch/SUEWS/Output_2000')

with this I got another keyerror

KeyError: 'Requested level (group) does not match index name (var)'

Can anyone please help me out; I am new to python

[sunt05]

Good to know the above issue on H5 is resolved. Please close this issue and I'll leave more comments on the saving issue in SuPy in a separate issue.