search

spedas / pyspedas

Python-based Space Physics Environment Data Analysis Software
https://pyspedas.readthedocs.io/
MIT License
154 stars 60 forks source link

cdf_to_tplot error loading Cluster data from CSA #716

Closed jameswilburlewis closed 7 months ago

jameswilburlewis commented 9 months ago

This code:

data = pyspedas.cluster.load_csa(probes=['C1','C2','C3'],trange=['2003-08-17/16:40', '2003-08-17/16:45'],datatypes='CP_CIS-HIA_ONBOARD_MOMENTS', time_clip=True)

causes a crash:

    325             ydata[ydata == var_atts["FILLVAL"]] = np.nan
    326     elif var_properties['Data_Type_Description'][:7] == 'CDF_INT':
    327         # NaN is only valid for floating point data
    328         # but we still need to handle FILLVAL's for
    329         # integer data, so we'll just set those to 0
--> 330         ydata[ydata == var_atts["FILLVAL"]] = 0
    332 tplot_data = {'x': xdata, 'y': ydata}
    334 # Data may depend on other data in the CDF.

TypeError: 'numpy.int32' object does not support item assignment

I think the issue is that ydata is actually a scalar for this variable...I'm not sure how that even happens in a CDF (would expect a 1-element array?), yet there it is.

jameswilburlewis commented 7 months ago

Possibly already fixed in either cdflib or cdf_to_tplot -- the error no longer occurs. I have re-enabled this test in the Cluster test suite.

jameswilburlewis commented 3 months ago

I think I figured out how we were getting bare scalars in Cluster CDFs. Something seems to be going wrong in their CDF creation process...If there is only a single timestamp, the associated data variables can lose their leading (time) dimension, and also get marked non-record-variant. That seems to be what happened here...a one-element array of scalars got turned into a single bare scalar. This should be more fully fixed in pytplot-mpl-temp 2.2.36.