Large files are not processed by CMOR

NOAA-GFDL / fre-python-tools

Python-based tools and interfaces to be used by FRE workflows

GNU Lesser General Public License v3.0

1 stars 2 forks source link

Large files are not processed by CMOR #16

Open sernik-59 opened 1 year ago

sernik-59 commented 1 year ago

I have a problem with processing large file hur from /archive/oar.gfdl.cmip6/ESM4/DECK/ESM4_historical_D1/gfdl.ncrc4-intel16-prod-openmp/pp/atmos_cmip/ts/daily/5yr/ having size ~7 GB. CMOR library gives error Error: cannot allocate memory for 1797552000 float tmp elts var 'hur' (table: day)

Probably it happens only on workstation. I have to try it on analysis, may be memory is bigger there.

aradhakrishnanGFDL commented 1 year ago

Hi @sernik-59 thanks for opening this issue. Looking at this https://cmor.llnl.gov/mydoc_appendix_A/, may be you can verify if cmor_setup is relevant to the problem, after checking on PPAN as well