search

spedas / pyspedas

Python-based Space Physics Environment Data Analysis Software
https://pyspedas.readthedocs.io/
MIT License
143 stars 58 forks source link

Crash in cdf_to_tplot for nonexistent DEPEND_0 variable #841

Closed jameswilburlewis closed 2 months ago

jameswilburlewis commented 2 months ago

See: https://github.com/spedas/pyspedas/issues/840

DE2 IDM CDFs have bad DEPEND_0 attributes for some variables. This currently causes a crash in cdf_to_tplot:

25-Apr-24 17:23:59: Downloading remote index: https://spdf.gsfc.nasa.gov/pub/data/de/de2/plasma_idm/vion250ms_cdaweb/1983/
25-Apr-24 17:24:00: File is current: de2_data/plasma_idm/vion250ms_cdaweb/1983/de2_vion250ms_idm_19830210_v01.cdf
Traceback (most recent call last):
  File "/Users/jwl/PycharmProjects/pyspedas/venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3550, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-14-c1824b9c4650>", line 3, in <module>
    idm_vars = pyspedas.de2.idm(trange=['1983-02-10', '1983-02-11'])
  File "/Users/jwl/PycharmProjects/pyspedas/pyspedas/de2/__init__.py", line 358, in idm
    tvars = load(instrument='idm', trange=trange, datatype=datatype, suffix=suffix, get_support_data=get_support_data, varformat=varformat, varnames=varnames, downloadonly=downloadonly, notplot=notplot, time_clip=time_clip, no_update=no_update)
  File "/Users/jwl/PycharmProjects/pyspedas/pyspedas/de2/load.py", line 139, in load
    tvars = cdf_to_tplot(out_files, suffix=suffix, get_support_data=get_support_data, varformat=varformat, varnames=varnames, notplot=notplot)
  File "/Users/jwl/PycharmProjects/pyspedas/venv/lib/python3.9/site-packages/pytplot/importers/cdf_to_tplot.py", line 257, in cdf_to_tplot
    = cdf_file.varinq(x_axis_var).Data_Type_Description
  File "/Users/jwl/PycharmProjects/pyspedas/venv/lib/python3.9/site-packages/cdflib/cdfread.py", line 217, in varinq
    vdr_info = self.vdr_info(variable)
  File "/Users/jwl/PycharmProjects/pyspedas/venv/lib/python3.9/site-packages/cdflib/cdfread.py", line 466, in vdr_info
    raise ValueError(f"Variable name '{variable}' not found.")
ValueError: Variable name 'Epoch_vel' not found.

We should check that x_axis_var exists before calling varinq on it, so we can avoid the exception, log a sensible error message, and move on to the next variable.

jameswilburlewis commented 2 months ago

A possible fix might be to skip the variable if x_axis_var is not in load_cdf_variables.

Also, some master CDFs seem to set Variable_Type to "Ignore Data" for problem variables -- we should probably check that case too.

See commented-out test_load_idm_data in de2/tests/tests.py

jameswilburlewis commented 2 months ago

Missing x_axis_var now skips the variable instead of crashing. Ignoring var_type=='ignore_data' was already implemented. The DE2 IDM example now works, and that test has been enabled in the DE2 test suite.