Support for AVIRIS-NG Data

[lewismc]

Hi Folks, we are currently working with AVIRIS-NG data and would really like to use spectral if possible. Right now when I load data I am having issues as outlined below. You can access the L1B and L2 data products here, ftp://avng.jpl.nasa.gov/AVNG_2015_data_distribution/. Can someone else confirm that they cannot load AVIRIS-NG products?

>>> import spectral as spy
>>> image = spy.open_image('ang20150422t163638_rdn_v1e_img')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/lmcgibbn/miniconda3/lib/python3.5/site-packages/spectral/spectral.py", line 490, in open_image
    raise IOError('Unable to determine file type or type not supported.')
OSError: Unable to determine file type or type not supported.
>>> exit()

If this is the case then we would like to write the functionality to work with AVIRIS-NG images and we will submit a pull request for this.

[lewismc]

I should have probably said, using the ENVI API and example displayed at http://www.spectralpython.net/fileio.html#envi-headers, I can read all of the Level 1B (orthocorrected radiance) and Level 2 (orthocorrected and atmospherically corrected reflectance data. I am just unable to use the AVIRIS API as documented at http://www.spectralpython.net/fileio.html#aviris

[tboggs]

That appears to be an ENVI-formatted file, as opposed to a "traditional" AVIRIS file. Instead of opening the data file directly, try opening it via the header file like this:

>>> image = spy.open_image('ang20150422t163638_rdn_v1e_img.hdr')

[tboggs]

@lewismc just now got your second comment. If I understand your comment, you are able to open the data files but you have to use spy.envi.open on the header instead of using the genericspy.open_image on the data file. Is that right? If so, I suppose I could add some logic to open_image that checks for an associated .hdr file and tries opening it with spy.envi.open.

[lewismc]

If I understand your comment, you are able to open the data files but you have to use spy.envi.open on the header instead of using the genericspy.open_image on the data file. Is that right?

Yes that's correct.

If so, I suppose I could add some logic to open_image that checks for an associated .hdr file and tries opening it with spy.envi.open.

I think there is more to it that this, take the following for example...

lmcgibbn@LMC-056430 ~/Desktop/AVNG_2015_data_distribution/L1B/ang20150422t163638_rdn_v1e $ python
Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:52:12)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import spectral as spy
>>> img = spy.aviris.open('ang20150422t163638_ort_igm.hdr')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/lmcgibbn/miniconda3/lib/python3.5/site-packages/spectral-0.18-py3.5.egg/spectral/io/aviris.py", line 78, in open
spectral.io.spyfile.InvalidFileError: File size not consistent with AVIRIS format.
>>> img = spy.aviris.open('ang20150422t163638_rdn_v1e_img')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/lmcgibbn/miniconda3/lib/python3.5/site-packages/spectral-0.18-py3.5.egg/spectral/io/aviris.py", line 78, in open
spectral.io.spyfile.InvalidFileError: File size not consistent with AVIRIS format.

It looks like we need to implement alternative logic for the AVIRIS-NG L1B and L2 file size(s).

On another note, the initial hdr file ang20150422t163638_ort_igm.hdr can be opened as follows

>>> hdr = spy.envi.open('ang20150422t163638_ort_igm.hdr')
>>> hdr.names[:5]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'BilFile' object has no attribute 'names'
>>> print(hdr)
    Data Source:   './ang20150422t163638_ort_igm'
    # Rows:           4138
    # Samples:         598
    # Bands:             3
    Interleave:        BIL
    Quantization:  64 bits
    Data format:   float64
>>> arr = hrd.load()
>>> arr.__class__
<class 'spectral.spectral.ImageArray'>
>>> print(arr.info())
    # Rows:           4138
    # Samples:         598
    # Bands:             3
    Data format:   float32
>>> arr.shape
(4138, 598, 3)

[tboggs]

I'm not seeing the "more to this" part. Historically, AVIRIS data were distributed in "raw" flightlines, where each line/row of data would have 514 samples/columns. If the entire flightline was in a single file, the file would have nrows * 614 pixels or if it was broken up into segments, each segment would have 512 * 614 pixels except for the final segment, which would have (nrows % 512) * 614 pixels. Since AVIRIS used 16-bit data values and has 224 bands, each row would be 275072 bytes. So aviris.open checks to see if the file size is a multiple of 275072. If it isn't, then it's not a "classic" AVIRIS file.

The "problem" is that more recently, they started georeferencing the images an rotating them north-up (it still bugs me that they do this). As a result they have to pad the raster with null values and the number of pixels per row is not equal to the actual flightline. But these newer files have ENVI headers that specify the file layout.

With regard to your examples:

>>> img = spy.aviris.open('ang20150422t163638_ort_igm.hdr')

That fails because aviris.open is for opening AVIRIS flightline files (not ENVI headers).

>>> img = spy.aviris.open('ang20150422t163638_rdn_v1e_img')

That fails for the reasons described above - the file size is consistent with a classic AVIRIS flightline file and there is no way to tell from the data alone what the image dimensions are.

>>> hdr = spy.envi.open('ang20150422t163638_ort_igm.hdr')

That works because you are using spectral.envi.open to open the file via it's associated ENVI header.

I don't know what you are trying to do here:

>>> hdr.names[:5]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'BilFile' object has no attribute 'names'

If you are trying to access header metadata, that would be via the hdr.metadata dict (note that the headers I browsed via your link don't actually have a "names" parameter, though some do have "band names").

So as I suggested, I think it wouldn't hurt to make spy.open_image a little smarter by looking to see if there is a ".hdr" file associated with a given data file. I'm not sure that changing aviris.open makes sense since it's a more specific function (and is wrapped by spy.open_image anyway).

Let me know what you think or if I've misunderstood something.

[lewismc]

Hi @tboggs ... slight delay sorry about that. If I open the AVIRIS and AVIRIS-NG images using the open_image function with the .hdr file passed as the argument, everything is working well. I can visualize and explore all of the data. Thank you very much for your input.