spedas / pyspedas

Python-based Space Physics Environment Data Analysis Software
https://pyspedas.readthedocs.io/
MIT License
147 stars 58 forks source link

tplot_restore: problem processing THEMIS L1 spin fit variable #951

Closed jameswilburlewis closed 1 month ago

jameswilburlewis commented 1 month ago

I'm seeing entries in the test logs like this, while loading THEMIS L1 spin fits:

2024-07-29T02:43:30.8508007Z 29-Jul-24 02:43:30: store_data: Could not create coordinate v1_dim for variable tha_fit
2024-07-29T02:43:30.8509466Z 29-Jul-24 02:43:30: store_data: ValueError exception text: conflicting sizes for dimension 'v1_dim': length 5 on <this-array> and length 2 on {'time': 'time', 'v1_dim': 'v1'}

So store_data is throwing an exception because of some confusion between data array sizes and DEPEND_N values. I don't see any log messages from cdf_to_tplot, and skteditor doesn't have any complaints about tha_fit, so maybe there's an issue with the logic in store_data.

Each sample of tha_fit is a 2x5 array. First dimension is E vs B, second dimension is the five outputs from the onboard spin fitter.

The exception is being caught within store_data, and the variable does get created. What it's complaining about is the creation of pandas or xarray coordinates for the data frame. I'm not even sure we use those anymore (it looks like they're there for spectrogram plotting).

jameswilburlewis commented 1 month ago

There's code in tplot_restore to transpose the data arrays as they're read in. There should have also been code to swap the V1, V2, V3 arrays to keep them lined up correctly with the data arrays. It's unclear at the moment what should happen if V1, V2, V3 themselves are multidimensional, as can happen with time-varying spectrogram bins, or spectrogram bins specified as upper/lower boundaries, as seen in some ERG data types.

The V1, V2, V3 swapping is now implemented, fixing the immediate bug. I'll keep this task open until the results of multidimensional V1, V2, V3 can be validated.

jameswilburlewis commented 1 month ago

Loading an ESA spectrogram variable via tplot_restore seems to give the correct orientation of the time-dependent v1 bin centers.