Spectral library resampling question

[pythonic2020]

Hello, I am trying to resample an ENVI spectral library to the wavelengths and bandpass of a sensor.

Input speclib ("envi"): 480 bands covering 0.2 - 2.9 um, micron units Output speclib ("swiris"): 100 bands covering 0.96 - 2.44 um, nm units

There are bad values in the input library, so I tried setting the data ignore value thus,

envi.metadata['data ignore value'] = -1.2300000e+34

but that didn't appear to have any effect. I don't know how to set those values to NaN in a SpectralLibrary object.

I converted the BandInfo units of the output to microns:

# Convert swiris BandInfo to micron units to match envi speclib
swiris_bandinfo.centers = [x / 1000. for x in swiris_bandinfo.centers]

swiris_bandinfo.bandwidths = [x / 1000. for x in swiris_bandinfo.bandwidths]

I then tried to create BandResmpler object:

envi_swir2020_resampler = spy.BandResampler(envi.bands, swiris_bandinfo)

I then tried to apply the resampler, but I couldn't find documentation on how to do so. I tried this:

envi_swiris = envi_swir2020_resampler(envi.spectra)

Error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-88-17a4918efcc2> in <module>
----> 1 envi_swiris = envi_swir2020_resampler(envi.spectra)

~\miniconda3\envs\py38\lib\site-packages\spectral\algorithms\resampling.py in __call__(self, spectrum)
    236         Any target bands that do not have at lease one overlapping source band
    237         will contain `float('nan')` as the resampled band value.'''
--> 238         return np.dot(self.matrix, spectrum)

<__array_function__ internals> in dot(*args, **kwargs)

ValueError: shapes (100,480) and (439,480) not aligned: 480 (dim 1) != 439 (dim 0)

Could you make any suggestions as to what I am doing wrong here? Thank you in advance....

[tboggs]

The "data ignore value" metadata field is not used. If you want to remove the bands from the spectral library, you would have to do surgery on the opened image by first modifying the data (by removing the bad columns from the "spectra" attribute) , then modifying the "bands" attribute to remove the associated band centers and bandwidths.

If you wanted to replace NaN values in the library with some constant (e.g., zero), you could probably do that like this:

slib.spectra[np.isnan(slib.spectra)] = 0

Regarding the resampling error, the documentation states that the BandResampler should receive a single spectrum as input. So you could do it like this:

envi_swiris = np.array([envi_swir2020_resampler(s) for s in envi.spectra])

That said (and despite the documentation), according to the code, it looks like you should actually be able to pass multiple spectra in a single array if you have them in columns of the input. So this should work:

envi_swiris = envi_swir2020_resampler(envi.spectra.T).T

[pythonic2020]

Thank you! I was pretty close but that 439 dim in the error threw me off.

ENVI plotting somehow doesn't plot those bad values even though the "data ignore value" isn't set for the speclib. Perhaps it's hard-coded in ENVI? I'm looking into a way to quickly set those bad values to nan.... Anyway, I can just set plt.ylim(0, 1) and the plots look pretty good....

[pythonic2020]

This seems to have done the trick:

envi.spectra[18]
>>array([-1.2300000e+34, -1.2300000e+34,  1.7548670e-01,  1.9984178e-01,
        2.1063155e-01,  2.1460478e-01,  2.1685915e-01,  2.1990111e-01,
        2.2058058e-01,  2.2873949e-01,  2.3561078e-01,  2.4390015e-01,
        2.5014159e-01, -1.2300000e+34, -1.2300000e+34, -1.2300000e+34,
       -1.2300000e+34,  2.9385823e-01,  3.0357128e-01,  3.1490511e-01,...

envi.spectra[envi.spectra == -1.2300000e+34] = np.nan

envi.spectra[18]
>>array([       nan,        nan, 0.1754867 , 0.19984178, 0.21063155,
       0.21460478, 0.21685915, 0.21990111, 0.22058058, 0.22873949,
       0.23561078, 0.24390015, 0.2501416 ,        nan,        nan,
              nan,        nan, 0.29385823, 0.30357128, 0.3149051 ,
       0.32507223, 0.33639374, 0.34811777, 0.35825697, 0.3682927 ,
       0.37476093, 0.38296247, 0.38948968, 0.3960724 , 0.4032488 ,...

[pythonic2020]

Before, with bad values, no axis limits set:

After, with helpful nans, no axis limits set:

[pythonic2020]

Note that changing those bad values to NaN causes a problem with the resampling. Not sure what yet. Better to resample without NaNs involved, and change to NaNs later to make nice plots.

[pythonic2020]

Is there a quick way to copy a SpectralLibrary object? If I do:

new_speclib = old_speclib

the new one just references the old, so any changes I make to new also happen on old. I need a way to make a real copy of the object, like new = old.copy() in pandas. Any ideas? I already have a workaround, but a copy feature would be nice.

[tboggs]

Python is a copy-by-reference language so saying a = b just points variable a to whatever b is.

There isn't currently a copy method for spectral libraries. A relatively safe (but inefficient) way to do it is to save the library and load it into a new variable. A faster (but potentially not future-proof) way is to do this:

new_speclib = SpectralLibrary(np.array(old_speclib.spectra), header=old_speclib.metadata)

[pythonic2020]

Perfect! That is a very important function you should consider adding to the documentation if you haven't already.

Code I used:

import spectral as spy
new_speclib = spy.envi.SpectralLibrary(np.array(old_speclib.spectra), header=old_speclib.metadata)

When you say "potentially not future-proof" do you mean that you may change that functionality in the future yourself, and/or that Python/numpy changes could make that method obsolete?

[tboggs]

I meant the former, since copying the library this way relies on the internals of how the SpectralLibrary is set up. But it is unlikely that it would change any time soon so I think you're safe here.

[pythonic2020]

Sounds great, thanks again!