pysat / pysatMissions

Mission planning instrument tools for pysat
BSD 3-Clause "New" or "Revised" License
11 stars 1 forks source link

Trouble Exporting to netcdf #98

Open JonathonMSmith opened 2 years ago

JonathonMSmith commented 2 years ago

Describe the bug After creating an sgp4 instrument with missions I have tried to save to netcdf. When I use the to_netcdf function I get the error

AttributeError: NetCDF: Not a valid data type or _FillValue type mismatch

I tried saving a cnofs ivm instrument with to_netcdf and got a different error, but I was still unable to save it.

I'm wondering if this is because I'm not setting the kwargs correctly.

To Reproduce

Python 3.9.13 (main, Aug 25 2022, 18:29:29) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pysat

In [2]: sat1 = pysat.Instrument('missions', 'sgp4')

In [3]: sat1.load(2018,1)
<ipython-input-3-46da04f1f5b8>:1: DeprecationWarning: Meta now contains a class for global metadata (MetaHeader). Default attachment of global attributes to Instrument will be Deprecated in pysat 3.2.0+. Set `use_header=True` in this load call or on Instrument instantiation to remove this warning.
  sat1.load(2018,1)

In [4]: sat1.to_netcdf4('/Users/jmsmit37/sat1_30_degrees.nc')
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In [4], line 1
----> 1 sat1.to_netcdf4('/Users/jmsmit37/sat1_30_degrees.nc')

File ~/anaconda3/envs/missions/lib/python3.9/site-packages/pysat/_instrument.py:3618, in Instrument.to_netcdf4(self, fname, base_instrument, epoch_name, zlib, complevel, shuffle, preserve_meta_case, export_nan, unlimited_time, modify)
   3615 inst = self if modify else self.copy()
   3617 # Write the output file
-> 3618 pysat.utils.io.inst_to_netcdf(inst, fname=fname,
   3619                               base_instrument=base_instrument,
   3620                               epoch_name=epoch_name, zlib=zlib,
   3621                               complevel=complevel, shuffle=shuffle,
   3622                               preserve_meta_case=preserve_meta_case,
   3623                               export_nan=export_nan,
   3624                               unlimited_time=unlimited_time)
   3626 return

File ~/anaconda3/envs/missions/lib/python3.9/site-packages/pysat/utils/io.py:1816, in inst_to_netcdf(inst, fname, base_instrument, epoch_name, mode, zlib, complevel, shuffle, preserve_meta_case, check_type, export_nan, unlimited_time, meta_translation, meta_processor)
   1809 cdfkey = out_data.createVariable(epoch_name, 'i8',
   1810                                  dimensions=(epoch_name),
   1811                                  zlib=zlib,
   1812                                  complevel=complevel,
   1813                                  shuffle=shuffle)
   1815 # Attach epoch metadata
-> 1816 cdfkey.setncatts(export_meta[epoch_name])
   1818 # Attach the time index to the data
   1819 cdfkey[:] = (inst.index.values.astype(np.int64)
   1820              * 1.0E-6).astype(np.int64)

File src/netCDF4/_netCDF4.pyx:4408, in netCDF4._netCDF4.Variable.setncatts()

File src/netCDF4/_netCDF4.pyx:1694, in netCDF4._netCDF4._set_att()

File src/netCDF4/_netCDF4.pyx:1965, in netCDF4._netCDF4._ensure_nc_success()

AttributeError: NetCDF: Not a valid data type or _FillValue type mismatch

Expected behavior netcdf file is produced from the instrument

Desktop (please complete the following information):

Additional context All the dependencies for pysatMissions were installed from the git repo using pip install -r requirements.txt

apexpy and aacgmv2 were installed with pip

jklenzing commented 2 years ago

Linking to https://github.com/pysat/pysat/issues/585

Adding output file tests in the core code will help maintain support.

jklenzing commented 1 year ago

If you add the epoch_name='time', it works. I think this is a bug in pysat, since it assumes the epoch_name in the instrument is 'Epoch', but will not let you write with that name.

jklenzing commented 1 week ago

lol, just ran into this bug again.