search

E3SM-Project / e3sm-unified

A metapackage for a unified anaconda environment for analyzing results from the Energy Exascale Earth System Model (E3SM).
BSD 3-Clause "New" or "Revised" License
8 stars 8 forks source link

e3sm-unified no longer supports high res analysis (cdf5 not supported) #42

Closed vanroekel closed 5 years ago

vanroekel commented 5 years ago

I just went back to running mpas-analysis on the high res coupled output on theta. using the latest e3sm-unified I get the following error

Traceback (most recent call last):
  File "./compute_transects.py", line 163, in <module>
    output=args.output_filename_pattern)
  File "./compute_transects.py", line 36, in compute_transport
    mesh = xr.open_dataset(mesh)
  File "/lus/theta-fs0/projects/ccsm/acme/tools/e3sm-unified/base/envs/e3sm_unified_1.2.2_py2.7_nox/lib/python2.7/site-packages/xarray/backends/api.py", line 320, in open_dataset
    **backend_kwargs)
  File "/lus/theta-fs0/projects/ccsm/acme/tools/e3sm-unified/base/envs/e3sm_unified_1.2.2_py2.7_nox/lib/python2.7/site-packages/xarray/backends/netCDF4_.py", line 332, in open 
    ds = opener()
  File "/lus/theta-fs0/projects/ccsm/acme/tools/e3sm-unified/base/envs/e3sm_unified_1.2.2_py2.7_nox/lib/python2.7/site-packages/xarray/backends/netCDF4_.py", line 231, in _open_netcdf4_group
    ds = nc4.Dataset(filename, mode=mode, **kwargs)
  File "netCDF4/_netCDF4.pyx", line 2126, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1368, in netCDF4._netCDF4._get_format
ValueError: format not supported by python interface

It seems like libnetcdf in this version doesn't support cdf5. I went back in versions as I know I've used mpas-analysis with e3sm-unified on theta before and the last version cdf5 worked was v1.2.0

xylar commented 5 years ago

@vanroekel, this should have been fixed almost a year ago: https://github.com/conda-forge/libnetcdf-feedstock/issues/42

You appear to be 2 versions behind in the version of e3sm-unified you're using. The latest e3sm-unified is 1.2.4 and it can be accessed by sourcing the script load_latest_e3sm_unified.sh, as I've detailed in two different emails to e3sm-all. The full path on cooley is:

/lus/theta-fs0/projects/ccsm/acme/tools/e3sm-unified/load_latest_e3sm_unified.sh

(Don't do analysis on theta or cooley login nodes -- they really don't like it!)

Modules are not supported by the latest conda (via an environment variable issue between proj4 and basemap) so that's why modules are no longer being updated.

vanroekel commented 5 years ago

Definitely not using modules or working on the login node. I was using e3sm-unified 1.2.3. Given this, I actually just realized that my error file was being appended to by multiple jobs and the error I posted is indeed old. The reason my job was dying had nothing to do with e3sm-unified. It was a bad file path further down in the file. I've also updated to 1.2.4. Things seem fine now. Sorry for the false alarm.

xylar commented 5 years ago

So glad!