search

raphaeldussin / freedompp

post-processing MOM6 history files
0 stars 1 forks source link

freedompp fails with `ValueError: buffer size must be a multiple of element size` #6

Open jetesdal opened 4 months ago

jetesdal commented 4 months ago

I want to extract ideal age variable (agessc) from downscaled annual z output. The component is named D2ocean_annual_z in the tar file. This is for the CM4X 1/8th historical run (CM4_historical_c192_OM4p125). I use freedompp on PP/AN as:

freedompp -t ts -f agessc -c D2ocean_annual_z -s 2010 -e 2010 -d /archive/Raphael.Dussin/FMS2019.01.03_devgfdl_20230608/CM4_historical_c192_OM4p125/gfdl.ncrc5-intel22-prod-openmp/history -o /archive/Jan-erik.Tesdal/FMS2019.01.03_devgfdl_20230608/CM4_historical_c192_OM4p125/gfdl.ncrc5-intel22-prod-openmp/pp

which results in the following error:

/nbhome/jet/miniconda3/bin/freedompp:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').run_script('freedompp==0.0.1', 'freedompp')
Traceback (most recent call last):
  File "/nbhome/jet/miniconda3/bin/freedompp", line 4, in <module>
    __import__('pkg_resources').run_script('freedompp==0.0.1', 'freedompp')
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/pkg_resources/__init__.py", line 722, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1561, in run_script
    exec(code, namespace, namespace)
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/freedompp-0.0.1-py3.9.egg/EGG-INFO/scripts/freedompp", line 185, in <module>
    write_timeserie(field, comesfrom, yearstart, yearend, **kwargs)
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/freedompp-0.0.1-py3.9.egg/freedompp/libfreedompp.py", line 134, in write_timeserie
    ds, fids = open_files_from_archives(
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/freedompp-0.0.1-py3.9.egg/freedompp/libIO.py", line 92, in open_files_from_archives
    ds = xr.open_mfdataset(open_files, **kwargs)
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/xarray-2024.6.0-py3.9.egg/xarray/backends/api.py", line 1054, in open_mfdataset
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/xarray-2024.6.0-py3.9.egg/xarray/backends/api.py", line 1054, in <listcomp>
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/xarray-2024.6.0-py3.9.egg/xarray/backends/api.py", line 571, in open_dataset
    backend_ds = backend.open_dataset(
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/xarray-2024.6.0-py3.9.egg/xarray/backends/scipy_.py", line 326, in open_dataset
    store = ScipyDataStore(
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/xarray-2024.6.0-py3.9.egg/xarray/backends/scipy_.py", line 178, in __init__
    scipy_dataset = _open_scipy_netcdf(
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/xarray-2024.6.0-py3.9.egg/xarray/backends/scipy_.py", line 126, in _open_scipy_netcdf
    return scipy.io.netcdf_file(filename, mode=mode, mmap=mmap, version=version)
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/scipy/io/_netcdf.py", line 279, in __init__
    self._read()
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/scipy/io/_netcdf.py", line 612, in _read
    self._read_var_array()
  File "/nbhome/jet/miniconda3/lib/python3.9/site-packages/scipy/io/_netcdf.py", line 724, in _read_var_array
    rec_array = frombuffer(self.fp.read(self._recs*self._recsize),
ValueError: buffer size must be a multiple of element size
raphaeldussin commented 4 months ago

it looks like it's using the old scipy.io as a backend, try installing netcdf4 it should work better

jetesdal commented 4 months ago

I have it installed

netCDF4 1.6.2

I also tried to use a clean env from scratch:

conda create --name freedompp python=3.10 
conda activate freedompp

git clone https://github.com/raphaeldussin/freedompp.git
cd freedompp

python setup.py install
conda install netCDF4

The error remains the same. ValueError: buffer size must be a multiple of element size