Some ERG specplots are displayed incorrectly

[jameswilburlewis]

A few ERG spectrum plots (not all!) are plotting with all vertical stripes, as if the energy bins all had the same values. They used to look OK in the past.

Example:

import pyspedas
from pytplot import tplot
pyspedas.erg.hep(trange=['2017-03-27', '2017-03-28'])
tplot(['erg_hep_l2_FEDO_L', 'erg_hep_l2_FEDO_H'])

The upper panel looks OK, the lower panel looks weird. For reference, I've included what it looked like before (erg_before), and what it looks like now (erg_now). I suppose it's possible that something has changed in the data files or the load routine, but I have a feeling something has changed with specplot.

I am not sure which version of pytplot was used to generate the older plot -- it's possible it was the old QT version. Still looking into that....

[jameswilburlewis]

Setting

pytplot.options('erg_hep_l2_FEDO_H', 'y_no_resample', 1)

seems to make it look like it did before. Still feels like a bug though?

[jimm02]

Ok, fixed this one, resample was wrong when the Y bin values for the data were not between yrange[0] and yrange[1] for the variable..

I'll need to check more spectra before closing

[jameswilburlewis]

Related issue: keograms don't plot at all (hangs forever) unless y_no_resample is set to 1.

It seems to be getting stuck here:

  File "/Users/jwl/PycharmProjects/PyTplot/pytplot/MPLPlotter/specplot.py", line 230, in specplot
    out_values1 = specplot_resample(out_values, vdata, vdata1)
  File "/Users/jwl/PycharmProjects/PyTplot/pytplot/MPLPlotter/specplot.py", line 48, in specplot_resample
    ss_ini = np.intersect1d(xxx[0], yyy[0])
  File "/Users/jwl/PycharmProjects/pyspedas/venv/lib/python3.9/site-packages/numpy/lib/arraysetops.py", line 445, in intersect1d
    ar2 = unique(ar2)
  File "/Users/jwl/PycharmProjects/pyspedas/venv/lib/python3.9/site-packages/numpy/lib/arraysetops.py", line 274, in unique
    ret = _unique1d(ar, return_index, return_inverse, return_counts,
  File "/Users/jwl/PycharmProjects/pyspedas/venv/lib/python3.9/site-packages/numpy/lib/arraysetops.py", line 336, in _unique1d
    ar.sort()

To reproduce:

import pyspedas
from pytplot import tplot
ask_vars = pyspedas.themis.ask(trange=['2013-11-05', '2013-11-06'])
tplot('thg_ask_atha')

This is happening with the latest version from github as of Feb 7.

[jimm02]

Hmm, It doesn't hang for me, but the plot looks bad.. All blue.

[jameswilburlewis]

It did eventually finish for me this time (previously, it crashed my python session....). All blue, like you said.

[jimm02]

The no_resample version is mostly black, different zranges, if you reset zrange minimum to zero, when not resampling, then you get the blue..

Pretty color, the blue

[jameswilburlewis]

I just pushed an update to specplot_resample that gets rid of the pretty, pretty blue in the ASK plot. The resampling process was missing the top index of the resampled data array (so it was left with 0 instead of the correct value), so I added a couple of assignments to ensure the upper and lower bins at each time got assigned. (Maybe only the upper one was needed?) Possibly related to that, I also see what looks like an out of bounds array index issue in the resampling loop, not fixed yet, just flagged in the comments because I'm not sure what the correct fix is.

Also, ChatGPT came up with an optimized version that avoids the np.intersection1d calls. That helped, but it's still a bit slow, though. I have the optimized version as the default now, but you can just comment out the assignment statement below it to revert to the original.

Anyway, please take a look, make sure I didn't break some other plot with my patches, and maybe take a close look at the array indexing. Is there ever a case where the bin values are non-monotonic at a single time index? I can't convince myself whether that would make sense or not....

[jameswilburlewis]

yyy = np.where(vdata_hi < vtmp[i+1]) # But i+1 is out of bounds on the last iteration>?

OK, I see that vtmp is defined with an extra element, so there's no out of bounds array indexing. But changing '<' to '<=' here and in a couple of other places seems like the right fix to the problem, so I've pushed that commit, and I think this resolves that problem. As for the slowness:

        fig_size = fig.get_size_inches()*fig.dpi
        ny = fig_size[1]*5 #maybe this will work better
        vdata1 = np.arange(0, ny, dtype=np.float64)*(ycrange[1]-ycrange[0])/(ny-1) + ycrange[0]

        out_values1 = specplot_resample(out_values, vdata, vdata1)
        out_values = out_values1

maybe we don't need that factor of 5 in the ny calculation anymore?