search

capstone-coal / pycoal

Python toolkit for characterizing Coal and Open-pit surface mining impacts on American Lands
http://capstone-coal.github.io/
GNU General Public License v2.0
28 stars 14 forks source link

process AVIRIS imagery #49

Closed ghost closed 7 years ago

ghost commented 7 years ago

this issue is to track progress on processing our new imagery. we can also use this thread to discuss the methodology of batch processing.

ghost commented 7 years ago

i will be using the USGS library as before while we await ASTER support.

i'm assuming we want to use the orthocorrected, scaled reflectance image files:

[ec2-user@ip-172-31-28-45 ~]$ find ~/AVNG_data_distribution/L2/ -name 'ang*_corr_*_img.hdr' | sort
/home/ec2-user/AVNG_data_distribution/L2/ang20140912t175835_rfl_v1c/ang20140912t175835_corr_v1c_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20140912t191051_rfl_v1c/ang20140912t191051_corr_v1c_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20140912t191744_rfl_v1c/ang20140912t191744_corr_v1c_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20140912t194636_rfl_v1c/ang20140912t194636_corr_v1c_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20140912t195442_rfl_v1c/ang20140912t195442_corr_v1c_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t153209_rfl_v1e/ang20150420t153209_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t155846_rfl_v1e/ang20150420t155846_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t160719_rfl_v1h/ang20150420t160719_corr_v1h_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t161618_rfl_v1h/ang20150420t161618_corr_v1h_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t162455_rfl_v1e/ang20150420t162455_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t164200_rfl_v1e/ang20150420t164200_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t165116_rfl_v1e/ang20150420t165116_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t170010_rfl_v1e/ang20150420t170010_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t170855_rfl_v1e/ang20150420t170855_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t171715_rfl_v1e/ang20150420t171715_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t172540_rfl_v1e/ang20150420t172540_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t173533_rfl_v1e/ang20150420t173533_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t174355_rfl_v1e/ang20150420t174355_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t181345_rfl_v1e/ang20150420t181345_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t182050_rfl_v1e/ang20150420t182050_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t182808_rfl_v1e/ang20150420t182808_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t183606_rfl_v1e/ang20150420t183606_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t184333_rfl_v1e/ang20150420t184333_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t185149_rfl_v1e/ang20150420t185149_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t185951_rfl_v1e/ang20150420t185951_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150420t190759_rfl_v1e/ang20150420t190759_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t161622_rfl_v1e/ang20150422t161622_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t161902_rfl_v1e/ang20150422t161902_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t162859_rfl_v1e/ang20150422t162859_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t163638_rfl_v1e/ang20150422t163638_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t164506_rfl_v1e/ang20150422t164506_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t165855_rfl_v1e/ang20150422t165855_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t170532_rfl_v1e/ang20150422t170532_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t171251_rfl_v1e/ang20150422t171251_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t171932_rfl_v1e/ang20150422t171932_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t172636_rfl_v1e/ang20150422t172636_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t173746_rfl_v1e/ang20150422t173746_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t174508_rfl_v1e/ang20150422t174508_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t175213_rfl_v1e/ang20150422t175213_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t175923_rfl_v1e/ang20150422t175923_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t180646_rfl_v1e/ang20150422t180646_corr_v1e_img.hdr
/home/ec2-user/AVNG_data_distribution/L2/ang20150422t181932_rfl_v1e/ang20150422t181932_corr_v1e_img.hdr

we will want to figure out a naming convention for the subdirectory (e.g., L3M) and the filename (e.g., ang*_class_img.hdr).

ghost commented 7 years ago

since there are 44 images, this would take quite a long time to process at 6–12 hours apiece. at that rate we would finish in 11 to 22 days.

i'm going to try to select the most interesting images and classify those first.

we should really see if we can optimize the time complexity of the mineral classification algorithm. i have discussed several possibilities.

this batch job should give us some baseline timing information. i plan to write the completion and running time of each image to a log file.

this effort will give some useful insight on how best to implement a classify multiple images method which seems like it will be important.

ghost commented 7 years ago

In order to figure out which images to process, it is necessary to go back to the AVIRIS data portal to look at the locations and the quick look images.

For convenience, here again are links to the AVIRIS-C Data Portal and AVIRIS-NG Data Portal. We might like to link to these on the wiki or the website.

The data on our server was initially identified on the mailing list. These are the flight logs:

Recall that additional possible imagery was found in Colorado and West Virginia, but the results were ambiguous for both the former and latter.

ghost commented 7 years ago

The first image I am interested in processing is the Craig Power Plant Coal Mine. The other San Juan images in flight ang20150422t are very similar to our original image. However, the following series of overlapping flight lines from flight ang20150420t give a broad coverage over the mine and the surrounding terrain: 4C_Tgt4_11, 4C_Tgt4_10, 4C_Tgt4_9, 4C_Tgt4_8, 4C_Tgt4_7, 4C_Tgt4_6, 4C_Tgt4_5, 4C_Tgt4_4, and 4C_Tgt4_3.

These files are on our server at the following locations respectively:

~/AVNG_data_distribution/L2/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img.hdr
~/AVNG_data_distribution/L2/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img.hdr
~/AVNG_data_distribution/L2/ang20150420t185951_rfl_v1e/ang20150420t185951_corr_v1e_img.hdr
~/AVNG_data_distribution/L2/ang20150420t184333_rfl_v1e/ang20150420t184333_corr_v1e_img.hdr
~/AVNG_data_distribution/L2/ang20150420t182808_rfl_v1e/ang20150420t182808_corr_v1e_img.hdr
~/AVNG_data_distribution/L2/ang20150420t181345_rfl_v1e/ang20150420t181345_corr_v1e_img.hdr
~/AVNG_data_distribution/L2/ang20150420t190759_rfl_v1e/ang20150420t190759_corr_v1e_img.hdr
~/AVNG_data_distribution/L2/ang20150420t185149_rfl_v1e/ang20150420t185149_corr_v1e_img.hdr
~/AVNG_data_distribution/L2/ang20150420t183606_rfl_v1e/ang20150420t183606_corr_v1e_img.hdr
~/AVNG_data_distribution/L2/ang20150420t182050_rfl_v1e/ang20150420t182050_corr_v1e_img.hdr

I will now write a script to process these 10 files which I estimate will take 5 days. Each image will be available as soon as it is done, so hopefully we will be able to start talking about them sooner.

ghost commented 7 years ago

Based on the quick look, the input data for 4C_Tgt4_6 might have some errors.

ghost commented 7 years ago

The messy results from my previous processing are now in ~/browtayl/pull-request-44. Files pertaining to this issue will be stored in ~/browtayl/issue-49.

ghost commented 7 years ago

@lewismc @heidiaclayton I am now processing the selected files with the script ~/browtayl/issue-49/classify.py:

#!/usr/bin/env python

# usage: ./classify.py >> classify.log 2>&1 &

import os
import time
import re
import sys
sys.path.append('/home/ec2-user/browtayl/pycoal')
import pycoal
import pycoal.mineral

# usgs digital spectral library 06
libraryFilename = '/home/ec2-user/USGS/s06av95a_envi.hdr'

# images identified here: https://github.com/capstone-coal/pycoal/issues/49#issuecomment-292400657
imageFilenames = [
    '/home/ec2-user/AVNG_data_distribution/L2/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t185951_rfl_v1e/ang20150420t185951_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t184333_rfl_v1e/ang20150420t184333_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t182808_rfl_v1e/ang20150420t182808_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t181345_rfl_v1e/ang20150420t181345_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t190759_rfl_v1e/ang20150420t190759_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t185149_rfl_v1e/ang20150420t185149_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t183606_rfl_v1e/ang20150420t183606_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t182050_rfl_v1e/ang20150420t182050_corr_v1e_img.hdr'
]

# save classified images in `~/AVNG_data_distribution/LC` directory with `_class` suffix
classifiedPattern = '(.*)L2/(.*/)(.*)(.hdr)'
classifiedDirectories = [re.sub(classifiedPattern,'\\1LC/\\2',filename) for filename in imageFilenames]
classifiedFilenames = [re.sub(classifiedPattern,'\\1LC/\\2\\3_class\\4',filename) for filename in imageFilenames]

# write completion and running time to log file
logFile = '/home/ec2-user/browtayl/issue-49/classify.log'

# instantiate the classifier
mc = pycoal.mineral.MineralClassification(libraryFilename)

# classify each image
for imageFilename,classifiedDirectory,classifiedFilename in zip(imageFilenames,classifiedDirectories,classifiedFilenames):

    # log start time
    with open(logFile, 'a') as log:
        log.write(str(int(time.time()))+' CLASSIFY '+imageFilename+'\n')

    # make directory
    os.mkdir(classifiedDirectory)

    # classify image
    mc.classifyImage(imageFilename, classifiedFilename)

    # log end time
    with open(logFile, 'a') as log:
        log.write(str(int(time.time()))+' SAVED '+classifiedFilename+'\n')

# log completion
with open(logFile, 'a') as log:
    log.write(str(int(time.time()))+' DONE.\n')

As you can see, I decided to save the files in an LC directory structure. Batch processing is slightly complicated by the fact that each image is in its own directory, so I used some regular expressions to help create them. It would be nice to find a more elegant way of doing this.

You can monitor the progress of the job by watching ~/browtayl/issue-49/classify.log which should be appended with start and end times and any messages from Spectral Python or Numpy. We can use these to calculate running time for now.

Hopefully we'll have results for at least one image in time for tomorrow's meeting.

lewismc commented 7 years ago

winning

ghost commented 7 years ago

some kind of progress meter might also be helpful here. i'll make a note of that in the logging issue.

ghost commented 7 years ago

@lewismc @heidiaclayton The first image took approximately 7 hours:

[ec2-user@ip-172-31-28-45 ~]$ cat browtayl/issue-49/classify.log 
1491536273 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img.hdr
1491580474 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img_class.hdr
1491580474 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img.hdr
/usr/local/lib/python2.7/site-packages/spectral/algorithms/algorithms.py:1323: RuntimeWarning: invalid value encountered in true_divide
  v = v / np.sqrt(v.dot(v))
[ec2-user@ip-172-31-28-45 ~]$ date -u -d "1491580474 seconds - 1491536273 seconds" +\"%H:%M\"
"07:11"

The classifier found regions of Schwertmannite BZ93-1 and some Renyolds_Tnl_Sludge SM93-15 in the vicinity of the Craig Power Plant Coal Mine. Some other interesting classes include Cyanide_Trihydrate CT1, Cyanide_Potassium_Ferro, Green_Slime SM93-14A Summitv, and Blck_Mn_Coat_Tailngs LV95-3. There seem to be far more unique classes in this image than the last few.

Python 2.7.9 (default, Jun 29 2016, 13:08:31) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> >>> >>> >>> 
>>> import spectral
>>> craigFilename = 'ang20140912t192359_corr_v1c_img_class.hdr'
>>> craig = spectral.open_image(craigFilename)
>>> craig
    Data Source:   '././ang20140912t192359_corr_v1c_img_class.img'
    # Rows:           4254
    # Samples:         872
    # Bands:             1
    Interleave:        BIP
    Quantization:  16 bits
    Data format:    uint16
>>> def indexSet(image):
...     return set(image.asarray().flatten().tolist())
... 
>>> indexSet(craig)
set([0, 256, 1026, 3, 517, 519, 665, 11, 597, 528, 530, 20, 22, 225, 537, 1050, 28, 541, 30, 543, 32, 545, 34, 347, 37, 550, 552, 41, 946, 1263, 48, 562, 513, 56, 572, 1085, 62, 575, 1088, 577, 1025, 581, 70, 1095, 974, 74, 589, 78, 79, 593, 595, 1109, 1111, 91, 96, 610, 107, 18, 110, 112, 114, 116, 117, 72, 121, 634, 635, 125, 640, 641, 1154, 643, 647, 649, 139, 194, 1166, 143, 659, 1172, 661, 662, 1175, 1096, 153, 1178, 155, 159, 1184, 161, 163, 165, 166, 1191, 681, 683, 689, 1205, 1206, 40, 699, 702, 1024, 706, 707, 228, 717, 209, 211, 1237, 726, 727, 1240, 1241, 1242, 1243, 223, 1249, 738, 1251, 1252, 230, 743, 233, 235, 749, 1262, 239, 1264, 753, 1266, 1267, 1269, 1270, 1272, 252, 254, 768, 770, 772, 261, 774, 776, 777, 780, 782, 301, 784, 742, 275, 788, 279, 792, 281, 794, 284, 287, 803, 300, 813, 307, 309, 310, 991, 317, 325, 737, 141, 1363, 1364, 342, 344, 740, 591, 655, 915, 350, 865, 867, 868, 869, 870, 872, 362, 364, 61, 1171, 887, 379, 894, 383, 897, 747, 1174, 902, 906, 911, 912, 403, 405, 406, 407, 679, 409, 751, 413, 415, 930, 440, 421, 1265, 939, 940, 754, 430, 944, 433, 434, 729, 948, 935, 950, 952, 957, 961, 967, 167, 972, 462, 976, 977, 979, 983, 985, 475, 988, 989, 990, 677, 480, 484, 81, 508, 1002, 1003, 494, 339, 500, 1013, 503, 1016, 1020, 1021, 1022, 511])
>>> 
>>> def classNameSet(image):
...     classNames = image.metadata.get('class names')
...     return map(lambda index: classNames[index], indexSet(image))
... 
>>> classNameSet(craig)
[u'No data', u'Dipyre BM1959;505.HLsp s06av95a=c', u'Hematite_Coatd_Qtz BR93-34C s06av95a=a', u'Actinolite HS22.3B s06av95a=b', u'Mizzonite BM1931;12 Scapolte s06av95a=c', u'Mizzonite HS350.3B HLSp Scap s06av95a=c', u'Praseodymium_Oxide GDS35 s06av95a=b', u'Adularia GDS57 Orthoclase s06av95a=bbb', u'Oligoclase HS110.3B s06av95a=c', u'Montmorillonite CM20 s06av95a=b', u'Montmorillonite CM26 s06av95a=b', u'Allanite HS293.3B s06av95a=b', u'Almandine HS114.3B Garnet s06av95a=b', u'Copiapite GDS21 s06av95a=b', u'Montmorillonite SCa-2.b s06av95a=b', u'Seawater_Coast_Chl SW1 s06av95a=a', u'Almandine WS477 Garnet s06av95a=b', u'Montmorillonite SWy-1 s06av95a=b', u'Almandine WS478 Garnet s06av95a=b', u'Mordenite GDS18 s06av95a=b', u'Almandine WS479 Garnet s06av95a=b', u'Mordenite+Clinopt. GDS151 s06av95a=b', u'Alunite AL706 Na__ s06av95a=b', u'Hematite GDS27 s06av95a=a', u'Alunite GDS82 Na82 s06av95a=b', u'Muscovite GDS108 s06av95a=b', u'Muscovite GDS111 Guatemal s06av95a=a', u'Alunite GDS84 Na03 s06av95a=a', u'Erionite+Merlinoite GDS144 s06av95a=c', u'Grass_dry.5+.5green AMX28 s06av95a=a', u'Alunite HS295.3B s06av95a=a', u'Muscovite GDS118 Capitan s06av95a=a', u'Microcline NMNH135231 Fspar s06av95a=c', u'Alunite SUSTDA-20 s06av95a=b', u'Muscovite IL107 s06av95a=a', u'Cyanide_Trihydrate CT1 s06av95a=a', u'Ammonio-Smectite GDS86 s06av95a=b', u'Natrolite HS169.3B Zeolite s06av95a=b', u'Cyanide_Potassium_Ferro s06av95a=a', u'Natrolite NMNH83380 Zeolite s06av95a=b', u'Hematite_Coatd_Qtz BR93-25B s06av95a=b', u'Nepheline HS19.3 s06av95a=c', u'Andesine HS142.3B s06av95a=c', u'Green_Slime SM93-14A Summitv s06av95a=a', u'Kaolin_Smect H89-FR-2 .5Kaol s06av95a=b', u'Andradite HS111.3B Garnet s06av95a=b', u'Nontronite NG-1.a s06av95a=b', u'Andradite WS474 s06av95a=b', u'Andradite WS487 Garnet s06av95a=b', u'Nontronite SWa-1.a s06av95a=b', u'Nontronite SWa-1.b <2um s06av95a=a', u'Paper_Cotton_Bond PAPR1 100% s06av95a=a', u'Plastic GGA-54 Grnhouse Roof s06av95a=a', u'Anorthite HS349.3B Plagio s06av95a=c', u'Antigorite NMNH96917.f <30u s06av95a=b', u'Olivine GDS71.b Fo91 <60um s06av95a=b', u'Arsenopyrite HS262.3B s06av95a=c', u'Albite HS143.3B Plagioclase s06av95a=c', u'Augite WS588 Pyroxene s06av95a=b', u'Augite WS592 Pyroxene s06av95a=b', u'Axinite HS342.3B s06av95a=b', u'Azurite WS316 s06av95a=a', u'Barite HS79.3B s06av95a=b', u'Andradite GDS12 Garnet s06av95a=c', u'Beryl GDS9 <150um gs s06av95a=b', u'Opal WS732 s06av95a=a', u'Orthoclase HS13.3B s06av95a=c', u'Biotite HS28.3B s06av95a=b', u'Palygorskite CM46a Attapulg s06av95a=b', u'Palygorskite PFL-1 Attapulg s06av95a=b', u'Renyolds_Tnl_Sludge SM93-15 s06av95a=a', u'Pectolite NMNH94865.a s06av95a=b', u'Perthite HS415.3B s06av95a=c', u'Phlogopite GDS20 fine fr s06av95a=b', u'Butlerite GDS25 s06av95a=a', u'Clinochlore GDS159 s06av95a=b', u'Aspen Aspen-1 Green-Top s06av95a=a', u'Bytownite HS106.3B Plagio s06av95a=c', u'Pinnoite NMNH123943 s06av95a=a', u'Aspen_Leaf-B DW92-3 s06av95a=a', u'Pitch_Limonite GDS104 Cu s06av95a=?', u'Polyhalite NMNH92669-4 s06av95a=a', u'Blue_Spruce DW92-5 needle s06av95a=a', u'Kerogen BK-Cornell s06av95a=a', u'Carnallite NMNH98011 s06av95a=b', u'Cheatgrass ANP92-11A s06av95a=a', u'Cassiterite HS279.3B s06av95a=c', u'Celsian HS200.3B s06av95a=d', u'Dry_Long_Grass AV87-2 s06av95a=a', u'Chalcedony CU91-6A s06av95a=a', u'Chalcopyrite HS431.3B s06av95a=b', u'Chalcopyrite S26-36 s06av95a=b', u'Chert ANP90-6D (White) s06av95a=?', u'Fir_Tree IH91-2 s06av95a=a', u'Pyrite HS35.3 s06av95a=b', u'Pyrite S142-1 s06av95a=c', u'Pyrolusite HS138.3B s06av95a=b', u'Juniper_Bush IH91-4B s06av95a=a', u'Lawn_Grass GDS91 (Green) s06av95a=a', u'Alunite GDS83 Na63 s06av95a=bb_', u'Pyrrhotite HS269.3B s06av95a=b', u'Quartz GDS74 Sand Ottawa s06av95a=c', u'Goethite_Thin_Film WS222 s06av95a=a', u'Rectorite ISR202 (RAr-1) s06av95a=?', u'Rectorite RAr-1 s06av95a=d', u'Cordierite HS346.3B s06av95a=d', u'Richterite HS336.3B Amphibol s06av95a=c', u'Clinoptilolite GDS152 Zeolit s06av95a=b', u'Clinoptilolite GDS2 Zeolite s06av95a=b', u'Maple_Leaves DW92-1 s06av95a=a', u'Roscoelite EN124 s06av95a=b', u'Rutile HS126.3B s06av95a=c', u'Pinon_Pine ANP92-14A needle s06av95a=a', u'Rabbitbrush ANP92-27 s06av95a=a', u'Russian_Olive DW92-4 s06av95a=a', u'Sage_Brush IH91-1B s06av95a=a', u'Cookeite CAr-1.c <30um s06av95a=b', u'Saltbrush ANP92-31A s06av95a=a', u'Saponite SapCa-1.AcB s06av95a=b', u'Tumbleweed ANP92-2C Dry s06av95a=a', u'Walnut_Leaf SUN (Green) s06av95a=a', u'Corrensite CorWa-1 s06av95a=b', u'Scolecite GDS7 Zeolite s06av95a=b', u'Covellite HS477.2B s06av95a=b', u'Cronstedtite M3542 s06av95a=b', u'Sepiolite SepSp-1 s06av95a=b', u'Grass_dry.4+.6green AMX27 s06av95a=a', u'Cuprite HS127.3B s06av95a=b', u'Grass_dry.6+.4green AMX29 s06av95a=a', u'Serpentine HS318.4B s06av95a=c', u'Grass_dry.8+.2green AMX31 s06av95a=a', u'Grass_dry.9+.1green AMX32 s06av95a=a', u'Lawn_Grass GDS91 +1shft 3nm s06av95a=a', u'Lawn_Grass GDS91 +const 1.0 s06av95a=a', u'Lawn_Grass GDS91 shifted 3nm s06av95a=a', u'Diopside HS317.3B (Cr)Pyx s06av95a=c', u'Diopside NMNHR18685 ~160 Pyx s06av95a=b', u'Spessartine NMNH14143 Garnet s06av95a=b', u'Spessartine WS480 Garnet s06av95a=b', u'Spessartine WS481 Garnet s06av95a=?', u'Dumortierite HS190.3B s06av95a=c', u'Sphalerite HS136.3B s06av95a=b', u'Sphalerite S102-7 s06av95a=c', u'Sphalerite S102-8 s06av95a=c', u'Sphene HS189.3B s06av95a=b', u'Spodumene HS210.3B s06av95a=c', u'Glauconite HS313.3B s06av95a=c', u'Staurolite HS188.3B s06av95a=b', u'Schwertmannite BZ93-1 s06av95a=b', u'Epidote GDS26.b <75um s06av95a=b', u'Stilbite HS482.3B Zeolite s06av95a=b', u'Epsomite GDS149 s06av95a=c', u'Sulfur GDS94 Reagent s06av95a=a', u'Eugsterite GDS140 Syn s06av95a=b', u'Syngenite GDS139 s06av95a=a', u'Fassaite HS118.3B s06av95a=c', u'Fluorapatite WS416 s06av95a=a', u'Thenardite GDS146 s06av95a=c', u'Gibbsite WS214 s06av95a=b', u'Thuringite SMR-15.d <30um s06av95a=a', u'Goethite HS36.3 s06av95a=b', u'Goethite MPCMA2-B FineGr adj s06av95a=b', u'Goethite MPCMA2-C M-Crsgrad2 s06av95a=b', u'Muscov+Jaros CU93-314 coatng s06av95a=b', u'Grossular HS113.3B-HCL Garnt s06av95a=d', u'Grossular WS485 Garnet s06av95a=b', u'Saponite SapCa-1 s06av95a=b', u'Bytownite HS105.3B Plagio s06av95a=b', u'Stonwll_Ply+.1grnGrass AMX33 s06av95a=a', u'Stonwll_Ply+.2grnGrass AMX34 s06av95a=a', u'Hedenbergite HS10.3B s06av95a=c', u'Hedenbergite NMNH119197 s06av95a=b', u'Sauconite GDS135 s06av95a=b', u'Nontronite NG-1.b <2um fr s06av95a=b', u'Phlogopite WS675 s06av95a=b', u'Calc.25+dolo.25+mont.5 AMX18 s06av95a=b', u'Hematite GDS69.b 104-150u s06av95a=b', u'Uvarovite NMNH106661 Garnet s06av95a=b', u'Vermiculite GDS13 Llano s06av95a=a', u'Vermiculite VTx-1.a <250 s06av95a=c', u'Vermiculite VTx-1.fls s06av95a=c', u'Vermiculite WS681 s06av95a=c', u'Vesuvianite HS446.3B Idocras s06av95a=a', u'Hematite HS45.3 s06av95a=b', u'Hematite WS161 s06av95a=b', u'Ammonio-Jarosite SCR-NHJ s06av95a=a', u'Aspen_Leaf-A DW92-2 s06av95a=a', u'Alun.33+Kaol.33+Musc.33 AMX2 s06av95a=a', u'Howlite GDS155 s06av95a=a', u'Alunite.5+MuscCU91-250A AMX4 s06av95a=a', u'Hydroxyl-Apatite WS425 s06av95a=b', u'Alunite0.5+Kaol_KGa-1 AMX3 s06av95a=b', u'Sepiolite SepNev-1.AcB s06av95a=a', u'Blackbrush ANP92-9A leaves s06av95a=a', u'Alunite_NH4+Jaro NMNH145596A s06av95a=b', u'Basalt_weathered BR93-43 s06av95a=b', u'Biotite-Chlorite_Mx BR93-36A s06av95a=b', u'Blue_Efflorscnt_Min SU93-300 s06av95a=b', u'Illite GDS4 (Marblehead) s06av95a=b', u'Illite IL101 (2M2) s06av95a=b', u'Illite IL105 (1Md) s06av95a=b', u'Illite IMt-1.a s06av95a=b', u'Psilomelane HS139.3B s06av95a=b', u'Illite IMt-1.b <2um s06av95a=a', u'Sepiolite SepSp-1.AcB s06av95a=b', u'Jadeite HS343.3B s06av95a=b', u'Jarosite GDS100 Na;Sy 90C s06av95a=a', u'Chlor+Goethite CU93-4B Phyl s06av95a=a', u'Kaolinite CM7 s06av95a=b', u'Jarosite GDS98 K 90C Syn s06av95a=a', u'Grass_dry.7+.3green AMX30 s06av95a=a', u'Chlorite_Phyllite CU91-238A s06av95a=b', u'Chlorite_Serpentine BR93-22B s06av95a=b', u'Serpentine HS8.3B s06av95a=c', u'Jarosite SJ-1 H3O;10-20% s06av95a=b', u'Dolomite.5+Na-mont.5 AMX21 s06av95a=b', u'Jarosite WS368 (Pb) s06av95a=c', u'Kainite NMNH83904 s06av95a=a', u'Rutile HS137.3B s06av95a=d', u'Erionite+Offretite GDS72 s06av95a=b', u'Chlorite+Muscovite CU93-65A s06av95a=a', u'Fe-Hydroxide SU93-106 amorph s06av95a=b', u'Goethite0.02+Quartz GDS240 s06av95a=a', u'Gyp.5+Kjar.3+hAlmsc.2 AMX24 s06av95a=a', u'Hematite.02+Quartz.98 GDS76 s06av95a=a', u'Kaol+Musc_intimate CU93-5C s06av95a=a', u'Chlorapatite WS423 s06av95a=b', u'Kaolin/Smect KLF511 .12Kaol s06av95a=b', u'Lazurite HS418.3B s06av95a=b', u'Kaolin_Smect H89-FR-5 .3Kaol s06av95a=b', u'Kaolin_Smect KLF506 .95Kaol s06av95a=b', u'Kaolin_Smect KLF508 .85Kaol s06av95a=b', u'Magnesite+Hydroma HS47.3B s06av95a=a', u'Magnetite_skarn BR93-5B s06av95a=b', u'Limonite HS41.3 s06av95a=b', u'Montmorillonite+Illite CM37 s06av95a=b', u'Montmorillonite+Illite CM42 s06av95a=b', u'Montmorillonite-Na CU93-52A s06av95a=c', u'Prochlorite SMR-14.c <30um s06av95a=a', u'Lizardite NMNHR4687.c 70 s06av95a=a', u'Maghemite GDS81 Syn (M-3) s06av95a=b', u'Andradite WS488 Garnet s06av95a=c', u'Microcline HS108.3B Feldsp s06av95a=bb_', u'Phlogopite_Sand_Mix BR93-20 s06av95a=a', u'Pyrite LV95-6A Weath on Tail s06av95a=b', u'Marialite NMNH126018-2 s06av95a=b', u'Hectorite SHCa-1 s06av95a=c', u'Meionite WS700.HLsp s06av95a=b', u'Teepleite+Trona NMNH102798 s06av95a=a', u'Microcline HS103.3B Feldspar s06av95a=c', u'Zincite+Franklin HS147.3B s06av95a=b', u'Blck_Mn_Coat_Tailngs LV95-3 s06av95a=b', u'Desert_Varnish ANP90-14 s06av95a=a', u'Desert_Varnish GDS141 s06av95a=a', u'Microcline HS82.3B Feldspar s06av95a=b']
>>> spectral.imshow(data=craig, classes=craig.read_band(0))
ImageView object:
  Display bands       :  [0]
  Interpolation       :  <default>
  RGB data limits     :
    R: [0.0, 1364.0]
    G: [0.0, 1364.0]
    B: [0.0, 1364.0]

>>> import numpy
>>> sludge = numpy.zeros([int(craig.metadata.get('classes')), 3], dtype=int)
>>> craig.metadata.get('class names').index('Schwertmannite BZ93-1 s06av95a=b')
742
>>> craig.metadata.get('class names').index('Renyolds_Tnl_Sludge SM93-15 s06av95a=a')
1154
>>> craig.metadata.get('class names').index('Renyolds_TnlSldgWet SM93-15w s06av95a=a')
1153
>>> sludge[1154] = numpy.asarray([255,0,0])
>>> sludge[742] = numpy.asarray([255,128,0])
>>> sludge[1153] = numpy.asarray([255,255,0])
>>> spectral.imshow(data=craig, classes=craig.read_band(0), colors=sludge)
ImageView object:
  Display bands       :  [0]
  Interpolation       :  <default>
  RGB data limits     :
    R: [0.0, 1364.0]
    G: [0.0, 1364.0]
    B: [0.0, 1364.0]

The second imshow command gives a color-coded result (sludge red, schwertmannite orange as before) which I saved on Google Drive. The results are a bit surprising because the strongest indications come from what appear in the quick look image to be fields adjacent to the power plant. The rectangular field appears to have been freshly worked, so maybe they dug up something in the soil. There are also concentrations around the ponds. The tailings class is not depicted in this image, but there are only a few pixels in the middle of the power facility.

Overall this image gives us some more indication that we can detect coal byproducts in mining regions, but we will want to spend more time interpreting the results.

ghost commented 7 years ago

At the same rate as the previous image, I estimate that each high-resolution scan over the San Juan mine will take approximately 28 hours to process:

> (define craig-size (* 4254 872))
> (define craig-time 7.2)
> (define sj-size (* 15862 909))
> (* craig-time (/ sj-size craig-size))
27.985969384454137

The first has taken 22 hours so far:

[ec2-user@ip-172-31-28-45 ~]$ cat browtayl/issue-49/classify.log 
1491536273 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img.hdr
1491580474 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img_class.hdr
1491580474 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img.hdr
/usr/local/lib/python2.7/site-packages/spectral/algorithms/algorithms.py:1323: RuntimeWarning: invalid value encountered in true_divide
  v = v / np.sqrt(v.dot(v))
[ec2-user@ip-172-31-28-45 ~]$ date -u -d "now - 1491580474 seconds" +\"%H:%M\"
"22:02"

I have created Issue https://github.com/capstone-coal/pycoal/issues/51 to eventually look into the performance. For now we will have to be content with results trickling in over the next week.

lewismc commented 7 years ago

Thanks for the update. If, at all possible, you are able to provide your opinion on where effciencies can be made and list them here it would be very helpful.

ghost commented 7 years ago

it's now been upwards of 56 hours, twice my original estimate:

[ec2-user@ip-172-31-28-45 ~]$ cat browtayl/issue-49/classify.log 
1491536273 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img.hdr
1491580474 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img_class.hdr
1491580474 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img.hdr
/usr/local/lib/python2.7/site-packages/spectral/algorithms/algorithms.py:1323: RuntimeWarning: invalid value encountered in true_divide
  v = v / np.sqrt(v.dot(v))
[ec2-user@ip-172-31-28-45 ~]$ echo $(( ($(date +%s) - 1491580474) / (60*60) ))
56

The running time is probably influenced not only by the number of pixels but by the proportion of no-data pixels. I do have what I need to get started on the mining id and environmental correlation, so I'm inclined to wait and see how long this takes. It may be worth our while to improve efficiency before classifying the rest of the images. We might also consider writing out the results periodically rather than accumulating them in memory until the very end.

ghost commented 7 years ago

It's worth mentioning that this one image is 25GB, much larger than our previous images:

[ec2-user@ip-172-31-28-45 ~]$ ls -l --block-size=GB /home/ec2-user/AVNG_data_distribution/L2/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img
-rwxr-xr-x 1 ec2-user ec2-user 25GB Mar 28 06:11 /home/ec2-user/AVNG_data_distribution/L2/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img
lewismc commented 7 years ago

Some of the images I've analyzed locally are >35GB...

ghost commented 7 years ago

It finally finished after nearly 70 hours, or almost 3 days:

[ec2-user@ip-172-31-28-45 ~]$ cat browtayl/issue-49/classify.log 
1491536273 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img.hdr
1491580474 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img_class.hdr
1491580474 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img.hdr
/usr/local/lib/python2.7/site-packages/spectral/algorithms/algorithms.py:1323: RuntimeWarning: invalid value encountered in true_divide
  v = v / np.sqrt(v.dot(v))
1491830486 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img_class.hdr
1491830486 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20150420t185951_rfl_v1e/ang20150420t185951_corr_v1e_img.hdr
[ec2-user@ip-172-31-28-45 ~]$ echo $(( (1491830486 - 1491580474) / (60*60) ))
69

It could be worse, but it could be better. It took 2.5 times longer than predicted, but that growth is still comfortably polynomial if not linear. The upshot is that this gives us more confidence that the classifier works even on very large images without crashing. It also provides quality metrics and added motivation for improving efficiency. I expect the remaining 8 images to take a similar amount of time, which would fill out the month if we don't speed things up. I will study the timing of the images we have processed so far in the performance issue, and I will continue to analyze the data here.

lewismc commented 7 years ago

This is brilliant. I'm going to make best efforts to have the SDS pipeline finished for out meeting this Friday. Can you please point me to some of the classified data products you've produced? Thank you

ghost commented 7 years ago

It looks like another one got classified after 43 hours:

[ec2-user@ip-172-31-28-45 ~]$ cat browtayl/issue-49/classify.log 
1491536273 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img.hdr
1491580474 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img_class.hdr
1491580474 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img.hdr
/usr/local/lib/python2.7/site-packages/spectral/algorithms/algorithms.py:1323: RuntimeWarning: invalid value encountered in true_divide
  v = v / np.sqrt(v.dot(v))
1491830486 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img_class.hdr
1491830486 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20150420t185951_rfl_v1e/ang20150420t185951_corr_v1e_img.hdr
1491988498 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20150420t185951_rfl_v1e/ang20150420t185951_corr_v1e_img_class.hdr
1491988498 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20150420t184333_rfl_v1e/ang20150420t184333_corr_v1e_img.hdr
[ec2-user@ip-172-31-28-45 ~]$ echo $(( (1491988498 - 1491830486) / (60*60) ))
43

I won't have time to analyze this till tomorrow.

@lewismc here should be our classified images:

[ec2-user@ip-172-31-28-45 ~]$ find . -name '*_class.hdr'
./AVNG_data_distribution/LC/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img_class.hdr
./AVNG_data_distribution/LC/ang20150420t185951_rfl_v1e/ang20150420t185951_corr_v1e_img_class.hdr
./AVNG_data_distribution/LC/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img_class.hdr
./browtayl/pull-request-44/f080702t01p00r08rdn_c_sc01_ort_img_class.hdr
./browtayl/pull-request-44/ang20150422t163638_corr_v1e_img_class.hdr
./browtayl/pull-request-44/f110609t01p00r11rdn_c_sc01_ort_img_class.hdr

In general, the new results can be found under the AVNG_data_distribution/LC directory and you can check browtayl/issue-49/classify.log for current processing status. keep in mind that the server is pretty much at capacity so there's not a lot of free cycles or memory.

lewismc commented 7 years ago

Thanks for the update

On Wed, Apr 12, 2017 at 11:46 AM Taylor Alexander Brown < notifications@github.com> wrote:

It looks like another one got classified after 43 hours.

[ec2-user@ip-172-31-28-45 ~]$ cat browtayl/issue-49/classify.log 1491536273 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img.hdr 1491580474 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img_class.hdr 1491580474 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img.hdr /usr/local/lib/python2.7/site-packages/spectral/algorithms/algorithms.py:1323: RuntimeWarning: invalid value encountered in true_divide v = v / np.sqrt(v.dot(v)) 1491830486 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img_class.hdr 1491830486 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20150420t185951_rfl_v1e/ang20150420t185951_corrv1e

img.hdr 1491988498 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20150420t185951_rfl_v1e/ang20150420t185951_corr_v1e_img_class.hdr 1491988498 CLASSIFY /home/ec2-user/AVNG_data_distribution/L2/ang20150420t184333_rfl_v1e/ang20150420t184333_corr_v1e_img.hdr [ec2-user@ip-172-31-28-45 ~]$ echo $(( (1491988498 - 1491830486) / (60*60) )) 43

I won't have time to analyze this till tomorrow.

@lewismc https://github.com/lewismc here should be our classified images:

[ec2-user@ip-172-31-28-45 ~]$ find . -name '*_class.hdr' ./AVNG_data_distribution/LC/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img_class.hdr ./AVNG_data_distribution/LC/ang20150420t185951_rfl_v1e/ang20150420t185951_corr_v1e_img_class.hdr ./AVNG_data_distribution/LC/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img_class.hdr ./browtayl/pull-request-44/f080702t01p00r08rdn_c_sc01_ort_img_class.hdr ./browtayl/pull-request-44/ang20150422t163638_corr_v1e_img_class.hdr ./browtayl/pull-request-44/f110609t01p00r11rdn_c_sc01_ort_img_class.hdr

In general, the results can be found under the LC directory and you can check browtayl/issue-49/classify.log for current processing status.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/capstone-coal/pycoal/issues/49#issuecomment-293671917, or mute the thread https://github.com/notifications/unsubscribe-auth/ABHJl1AzOYtwmKx88omBKtF6wV-bLYtuks5rvRwggaJpZM4Myy8s .

--

Lewis Dr. Lewis J. McGibbney Ph.D, B.Sc Director, MCMA Associates Phone: +1(626)498-3090 Skype: lewis.john.mcgibbney Email: lewis.mcgibbney@gmail.com

ghost commented 7 years ago

i analyzed some of our new results as described in the mining classification pull request. the mining classifier allows us to easily generate georeferenced classified files containing only the pixels we want to know about. when all the high-res san juan images have been processed we will have the most complete picture available.

ghost commented 7 years ago

Another image came in after 42 hours. There are still 5 images to go:

[ec2-user@ip-172-31-28-45 ~]$ cat browtayl/issue-49/classify.py  | grep 'hdr'
libraryFilename = '/home/ec2-user/USGS/s06av95a_envi.hdr'
    '/home/ec2-user/AVNG_data_distribution/L2/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t185951_rfl_v1e/ang20150420t185951_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t184333_rfl_v1e/ang20150420t184333_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t182808_rfl_v1e/ang20150420t182808_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t181345_rfl_v1e/ang20150420t181345_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t190759_rfl_v1e/ang20150420t190759_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t185149_rfl_v1e/ang20150420t185149_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t183606_rfl_v1e/ang20150420t183606_corr_v1e_img.hdr',
    '/home/ec2-user/AVNG_data_distribution/L2/ang20150420t182050_rfl_v1e/ang20150420t182050_corr_v1e_img.hdr'
classifiedPattern = '(.*)L2/(.*/)(.*)(.hdr)'
[ec2-user@ip-172-31-28-45 ~]$ cat browtayl/issue-49/classify.log  | grep 'SAVED'
1491580474 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20140912t192359_rfl_v1c/ang20140912t192359_corr_v1c_img_class.hdr
1491830486 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20150420t180624_rfl_v1e/ang20150420t180624_corr_v1e_img_class.hdr
1491988498 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20150420t185951_rfl_v1e/ang20150420t185951_corr_v1e_img_class.hdr
1492144680 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20150420t184333_rfl_v1e/ang20150420t184333_corr_v1e_img_class.hdr
1492298835 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20150420t182808_rfl_v1e/ang20150420t182808_corr_v1e_img_class.hdr

I estimate it will take 9 days to finish classifying the rest of the high-res San Juan Mine images:

>>> 5*43
215
>>> (5*43)/24.0
8.958333333333334
ghost commented 7 years ago

i found when investigating issue https://github.com/capstone-coal/pycoal/pull/67#issue-222380252 that the first high-res image, the 70-hour one, was processed incorrectly due to an unknown error. the result that every class in the image had the value 1. it should be possible to reclassify that image to see what might have gone wrong.

i expect several more images have been classified by now, so i'll provide updates when i examine them.

ghost commented 7 years ago

Three more images have been classified:

1492455372 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20150420t181345_rfl_v1e/ang20150420t181345_corr_v1e_img_class.hdr
1492613048 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20150420t190759_rfl_v1e/ang20150420t190759_corr_v1e_img_class.hdr
1492774106 SAVED /home/ec2-user/AVNG_data_distribution/LC/ang20150420t185149_rfl_v1e/ang20150420t185149_corr_v1e_img_class.hdr

Two more remain.

ghost commented 7 years ago

As before, I used our mining classifier like so:


>>> import sys
>>> sys.path.append('/media/study/OSU/spring17/cs463/coal')
>>> import pycoal
>>> import pycoal.mining
>>> mc = pycoal.mining.MiningClassification([u'Schwertmannite BZ93-1 s06av95a=b', u'Renyolds_TnlSldgWet SM93-15w s06av95a=a', u'Renyolds_Tnl_Sludge SM93-15 s06av95a=a'])
>>> mc.classifyImage('/media/study/OSU/spring17/cs463/data/AVNG_data_distribution/LC/ang20150420t181345_rfl_v1e/ang20150420t181345_corr_v1e_img_class.hdr', '/media/study/OSU/spring17/cs463/data/AVNG_data_distribution/LC/ang20150420t181345_rfl_v1e/ang20150420t181345_corr_v1e_img_class_mining.hdr')
Saving /media/study/OSU/fall16/cs461/data/AVNG_data_distribution/LC/ang20150420t181345_rfl_v1e/ang20150420t181345_corr_v1e_img_class_mining.img
>>> mc.classifyImage('/media/study/OSU/spring17/cs463/data/AVNG_data_distribution/LC/ang20150420t185149_rfl_v1e/ang20150420t185149_corr_v1e_img_class.hdr', '/media/study/OSU/spring17/cs463/data/AVNG_data_distribution/LC/ang20150420t185149_rfl_v1e/ang20150420t185149_corr_v1e_img_class_mining.hdr')
Saving /media/study/OSU/fall16/cs461/data/AVNG_data_distribution/LC/ang20150420t185149_rfl_v1e/ang20150420t185149_corr_v1e_img_class_mining.img
>>> mc.classifyImage('/media/study/OSU/spring17/cs463/data/AVNG_data_distribution/LC/ang20150420t190759_rfl_v1e/ang20150420t190759_corr_v1e_img_class.hdr', '/media/study/OSU/spring17/cs463/data/AVNG_data_distribution/LC/ang20150420t190759_rfl_v1e/ang20150420t190759_corr_v1e_img_class_mining.hdr')
Saving /media/study/OSU/fall16/cs461/data/AVNG_data_distribution/LC/ang20150420t190759_rfl_v1e/ang20150420t190759_corr_v1e_img_class_mining.img

I had to workaround the GDAL problem again like so:

$ history | grep gdal | tail 
 3100  which gdalwarp
 3101  export PATH=/media/study/OSU/winter17/cs462/gdal/gdal/gdal/apps:$PATH
 3102  gdalwarp -of envi ang20150420t181345_corr_v1e_img_class_mining.img ang20150420t181345_corr_v1e_img_class_mining_rot.img
 3104  gdalwarp -of envi ang20150420t185149_corr_v1e_img_class_mining.img ang20150420t185149_corr_v1e_img_class_mining_rot.img
 3106  gdalwarp -of envi ang20150420t190759_corr_v1e_img_class_mining.img ang20150420t190759_corr_v1e_img_class_mining_rot.img

Recall that this will not be necessary once I compile the upcoming GDAL release into QGIS.

The results are pretty cool so I'll be sharing them shortly in the environmental correlation issue.

ghost commented 7 years ago

The final two high-res San Juan Mine images have been processed, so now all ten images are done and the server resources are free.

The first image was apparently misclassified due to an unknown error. I will analyze the last two in the environmental correlation issue. Since initial processing is done I'm going to close this issue.

Additional processing, as with subset/threshold classification, can be tracked in future issues.