support ASTER

[ghost]

@lewismc @heidiaclayton this issue is to continue work from issue https://github.com/capstone-coal/pycoal/issues/18 and pull https://github.com/capstone-coal/pycoal/pull/44 to implement ASTER support for mineral classification. hopefully the additional samples will improve accuracy.

see commit https://github.com/browtayl/coal/commit/1e58d1237811dc8c46da493e3aeb9bb0fe819e9c in branch https://github.com/browtayl/coal/tree/ISSUE-36-mineral-classification-aster for my preliminary work on this. as discussed, we would like to implement a method to convert ASTER into a SpectralLibrary compatible format that can be consumed by the constructor. decisions will need to be made about how best to resample the library.

[lewismc]

ack

[ghost]

@lewismc @heidiaclayton we should consider incrementing the version number when this gets implemented so that we can tell our USGS-classified and ASTER-classified files apart.

[ghost]

@heidiaclayton @lewismc as discussed in our last meeting, i have bumped this to the new 0.3 milestone which is tentatively due 2017-04-21.

[ghost]

as mentioned, adding a method to convert the ASTER database to sqlite would be useful.

[heidiaclayton]

@browtayl @lewismc I'm planning on making another class inside mineral.py to implement ASTER. how does this sound?

[heidiaclayton]

Also, @browtayl , do you know if it's possible to create a general-purpose ASTER .hdr file, or does it have to be resampled for a specific image each time?

create_envi_spectral_library(spectrumIDs, bandInfo) Creates an ENVI-formatted spectral library for a list of spectra. Arguments: spectrumIDs (list of ints): List of SpectrumID values for of spectra in the “Spectra” table of the ASTER database. bandInfo (BandInfo): The spectral bands to which the original ASTER library spectra will be resampled. Returns: A SpectralLibrary object.

[heidiaclayton]

@browtayl I created an ENVI spectral library with the method you used here, however there a lot of no data values in the spectra.

aster_envi.zip

[ghost]

There are several possibilities here:

1) Make MineralClassification take an asterLibraryFilename parameter and convert the library to a suitable SpectralLibrary instance on instantiation. You have said that this conversion takes ~20 minutes which makes this option sound very inconvenient. 2) Add a method somewhere to convert a raw ASTER database file to a .sqlite file and add an asterDatabaseFilename parameter to MineralClassification to point to it. 3) Add a method somewhere to convert a raw ASTER database file to an ENVI spectral library file, a general-purpose .hdr as you say. Then the MineralClassification constructor could take the file name to an arbitrary ENVI spectral library with libraryFilename. 4) Add a method somewhere to convert a raw ASTER database file to a SpectralLibrary instance and do the resampling at instantiation. Same time constraints as before.

Personally I would lean towards (3). All you would have to do once you have converted ASTER into a SpectralLibrary instance is call its save(fileBaseName, description=None) method. So the method to create would have a signature like asterToEnvi(asterFilename, enviFilename). Since it would only be called once, you could make the case that it belongs either inside MineralClassification or in its own class.

Implement it however you think is easiest and most elegant. A working implementation is more persuasive than a hypothetical argument.

I don't know what to do about the No data values. They are currently handled correctly by the classification code, so it's probably not a big deal since they get mapped to zero. It probably does distort the spectral angle mapper algorithm somewhat, but that's more the algorithm's fault than the data's. A machine-learning algorithm would have an advantage here.

[ghost]

Regarding resampling, the image is resampled to the dimensions of the library at classification time, not vice versa. Currently with USGS the images are resampled down to 224 bands. With ASTER I chose 128 arbitrarily, in which case images would be resampled down to same. What we really want is an ideal tradeoff between dimensionality and accuracy, but who knows what that is.

[ghost]

Thanks for taking the lead on this component. Supporting the most extensive spectral library available will add huge value to this project, especially since I found in the subset and threshold issue that classifying with a full library gives better results than with a subset. Using ASTER would be an excellent choice for processing AVIRIS data en masse with XSEDE.

[ghost]

@heidiaclayton Currently reviewing your code. One thing I notice about the ASTER library is that although it contains more samples than the USGS library, it only includes a tiny subset from the USGS library. For example, schwertmannite is not in ASTER. Eventually we might want to be able to fully merge the two somehow, but for now ASTER processing is experimental.

[ghost]

Also we probably don't need to be searching for lunar soil lol.

[ghost]

And it appears that the many ancillary files don't contain spectra at all but extra information such as chemistry.

[ghost]

Implemented in pulls https://github.com/capstone-coal/pycoal/pull/76 and https://github.com/capstone-coal/pycoal/pull/77. Being able to convert an ASTER database to an ENVI file without digging around in the Spectral Python docs is super useful, but as previously mentioned extra work might be needed to combine it with the USGS database. I think we can save that for a future issue and consider this issue closed. Thanks @heidiaclayton for the addition.