search

spectralpython / spectral

Python module for hyperspectral image processing
MIT License
571 stars 139 forks source link

Unable to open current version of ASTER database #89

Closed RippleGylf closed 4 years ago

RippleGylf commented 5 years ago

I am trying to bring in the current version of the ASTER files available from https://speclib.jpl.nasa.gov/, but it runs into an IndexError on the first file every time. Is anyone aware of any changes to the formatting since the AsterDatabase.create function was made?

Error and traceback looks like this:

IndexError Traceback (most recent call last)

in () ----> 1 db = AsterDatabase.create("aster_lib.db", "HSI_DATA") /home/w036aat/.local/lib/python2.7/site-packages/spectral/database/aster.pyc in create(cls, filename, aster_data_dir) 235 db.cursor.execute(schema) 236 if aster_data_dir: --> 237 db._import_aster_files(aster_data_dir) 238 return db 239 /home/w036aat/.local/lib/python2.7/site-packages/spectral/database/aster.pyc in _import_aster_files(self, aster_data_dir) 284 print(('Importing %s.' % f)) 285 numFiles += 1 --> 286 sig = read_file(f) 287 s = sig.sample 288 if s['particle size'].lower == 'liquid': /home/w036aat/.local/lib/python2.7/site-packages/spectral/database/aster.pyc in read_file(filename) 111 if len(pair) < 2: 112 print(pair) --> 113 s.measurement[pair[0].lower()] = pair[1] 114 115 # Read signature spectrum IndexError: list index out of range
tboggs commented 5 years ago

Are you attempting to process the ASTER library or the ECOSTRESS library? I ask because the link you provided is for ECOSTRESS. Also, can you please provide the full output of the command you ran (it should have printed the name of the file and the pair of values it was attempting to process)?

RippleGylf commented 5 years ago

ECOSTRESS, as far as I can tell, appears to be the current manifestation of the ASTER library. I'm not exactly sure when this shift occurred, but any links to the ASTER library direct to this set of pages.

The output of the command I ran is as follows:

Importing HSI_DATA/mineral.sulfide.none.medium.vswir.s-7a.jpl.beckman.spectrum.txt. [u'']

tboggs commented 5 years ago

So this page indicates that ECOSTRESS was formerly the ASTER library. Give me some time to grab a few of the ECOSTRESS files and see how much effort would be involved in updating the import routine.

tboggs commented 5 years ago

I received a copy of the ECOSTRESS library. It appears the format header section of the data files has changed slightly so I will need to create a modified version of the import code to handle the changes.

tboggs commented 4 years ago

I've added the EcostressDatabase class, which is functionally identical to AsterDatabase but will parse ECOSTRESS library files instead.

In [2]: db = spy.EcostressDatabase.create('ecostress.db', './eco_data_ver1') 
...
Importing ./eco_data_ver1/mineral.silicate.inosilicate.coarse.vswir.in-8a.jpl.perkin.spectrum.txt.
Importing ./eco_data_ver1/vegetation.shrub.aloe.arborescens.tir.jpl077.jpl.nicolet.spectrum.txt.
Importing ./eco_data_ver1/mineral.silicate.phyllosilicate.coarse.tir.ps-12f.jpl.nicolet.spectrum.txt.
Processed 3403 files.

In [3]: db.print_query('SELECT COUNT() FROM Samples WHERE Name LIKE "%stone%"')
78