search

beatrixparis / connectivity-modeling-system

The CMS is a multiscale stochastic Lagrangian framework developed by Paris' Lab at the Rosenstiel School of Marine, Atmospheric & Earth Science to study complex behaviors, giving probabilistic estimates of dispersion, connectivity, fate of pollutants, and other Lagrangian phenomena. This repository facilitates community contributions to CMS modules
https://beatrixparis.github.io/connectivity-modeling-system/
GNU General Public License v3.0
31 stars 25 forks source link

NetCDF Error. NetCDF: Not a valid data type or _FillValue mismatch #38

Open croachutas opened 4 years ago

croachutas commented 4 years ago

I've recently installed CMS on a 'new' workstation (actually second hand hardware but a fresh install of Ubuntu 18.04.3). I compiled the various dependencies (HDF5, szip, zlib, netcdf C and netcdf fortran) from their source codes following instructions on the HDF5 and NetCDF websites before building CMS.

The test case works when set to ASCII output but switching it to netcdf output (mostly since I'd prefer less post-processing when running several million particles in my actual experiment) I get an error: Error in the netCDF file: expt_example/SCRATCH/traj_file_1.nc NetCDF: Not a valid data type or _FillValue type mismatch.

The same error occurs when running CMS with velocity fields converted to CMS format locally rather than downloaded remotely.

There are no issues when converting netCDF files to the CMS setup using getData, so this error seems to be exclusively related to how CMS writes NetCDF files out.

I'm currently using HDF5 v1.10.5, NetCDF C v4.7.2 and NetCDF Fortran v4.5.2

Doing a little googling the same error output has been reported with recent versions of netcdf-4python (https://github.com/Unidata/netcdf4-python/issues/929), and is apparently related to NetCDF-C 4.6.2 and onward being stricter than earlier versions about FillValue and Variable types matching.

I suspect it'd be possible to downgrade to an older version of NetCDF as a short term work around (or put up with the extra post-processing and HD-space demands from going with ASCII). However, for future compatibility you probably should double check the types you're assigning to the FIllValue...

croachutas commented 4 years ago

Can confirm downgrading to NetCDF C 4.6.1 and NetCDF Fortran 4.4.5 fixes this issue on my machine. Still, probably do wanna fix it properly in the CMS code to avoid it reoccurring for other people.

beatrixparis commented 4 years ago

Thank you Christopher, this is very helpful; we will look into a fix.

Claire

Sent from my iPhone

Claire Beatrix Paris-Limouzy, PhD Rosenstiel School of Marine and Atmospheric Science

De : Christopher Roach notifications@github.com Envoyé : Tuesday, December 3, 2019 5:29:31 AM À : beatrixparis/connectivity-modeling-system connectivity-modeling-system@noreply.github.com Cc : Subscribed subscribed@noreply.github.com Objet : Re: [beatrixparis/connectivity-modeling-system] NetCDF Error. NetCDF: Not a valid data type or _FillValue mismatch (#38)

Can confirm downgrading to NetCDF C 4.6.1 and NetCDF Fortran 4.4.5 fixes this issue on my machine. Still, probably do wanna fix it properly in the CMS code to avoid it reoccurring for other people.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbeatrixparis%2Fconnectivity-modeling-system%2Fissues%2F38%3Femail_source%3Dnotifications%26email_token%3DACZ27ZHJVJXAKYMGUE47XVTQWYYIXA5CNFSM4JN5FJO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFY4ECQ%23issuecomment-561103370&data=02%7C01%7Ccparis%40rsmas.miami.edu%7C0521106846aa4cdbd34408d777dbaf67%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637109657739228613&sdata=uinsr6sZ6dGZ8YGXeefiyHg21S4RTHkzYs0%2FIpFDWSk%3D&reserved=0, or unsubscribehttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACZ27ZEHQODSE2ERPB433ZLQWYYIXANCNFSM4JN5FJOQ&data=02%7C01%7Ccparis%40rsmas.miami.edu%7C0521106846aa4cdbd34408d777dbaf67%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637109657739233594&sdata=1IeeMMZfZvqU9dsTBLgM9fMbF8vBVicJbT12XlpxHfU%3D&reserved=0.

benfarmer05 commented 1 year ago

I am still receiving this error today, using an HPC with netcdf-c/4.7.3 and netcdf-fortran/4.5.2 installed. Outputting to ASCII solved the error.