update rookery imagery - Githubissues

7yl4r commented 1 year ago

Latest version of Rookery images ready from Digna in /datashare/regions/rookery/Processed/RookeryFinal/ReprocessedWithDEM/LastClassific/

steps:

[x] deleted 237 old images from bucket
[x] gsutil cp new images to bucket
[x] gbucket_to_gee_w_metadata_rookery.sh images from bucket to gee ImageCollection
[x] update gee module /users/tylarmurray/nerrs/classmap_helpers to point to latest ImageCollection

7yl4r commented 1 year ago

Logs from gsutil cp

(base) tylar@manglilloo:/srv/imars-objects/rookery/Processed/RookeryFinal/ReprocessedWithDEM/LastClassific$ gsutil cp *_Rookery_wDEM.tif gs://rookery-wv-classmaps

Copying file://20100301T162229_01_P009_WV02_ClassMap_vB2_Rookery_wDEM.tif [Content-Type=image/tiff]...
Copying file://20100301T162230_01_P010_WV02_ClassMap_vB1_Rookery_wDEM.tif [Content-Type=image/tiff]...
[...]
Copying file://20200929T162714_03_P006_WV03_ClassMap_vB2_Rookery_wDEM.tif [Content-Type=image/tiff]...
Copying file://20200929T162715_03_P007_WV03_ClassMap_vB2_Rookery_wDEM.tif [Content-Type=image/tiff]...
| [191 files][  3.0 GiB/  3.0 GiB]   20.5 MiB/s                                 
==> NOTE: You are performing a sequence of gsutil operations that may
run significantly faster if you instead use gsutil -m cp ... Please
see the -m section under "gsutil help options" for further information
about when gsutil -m can be advantageous.

Operation completed over 191 objects/3.0 GiB.

7yl4r commented 1 year ago

logs from gbucket transfer:

(base) tylar@manglilloo:~/wv-land-cover$ bash gee-uploads/gbucket_to_gee_w_metadata_rookery.sh rookery-wv-classmaps /srv/imars-objects/rookery/Processed/wv_ortho_xml/ users/tylarmurray/nerrs_rookery_v03 | tee rookery_upload-2023_07_28.log
checking if the collection users/tylarmurray/nerrs_rookery_v03 exists...
collection created.

*** Transfering file  20100301T162229_01_P009_WV02_ClassMap_vB2_Rookery_wDEM ***
*** parsing metadata...
{"dt_Y": "2010", "dt_m": "03", "dt_d": "01", "dt_H": "16", "dt_M": "22", "dt_S": "29", "number": "01", "pass_n": "009", "sat_n": "02", "adjustments_version": "vB2", "dt_a": "Mon", "dt_A": "Monday", "dt_w": "1", "dt_dd": "1", "dt_b": "Mar", "dt_B": "March", "dt_mm": "3", "dt_y": "10", "dt_HH": "16", "dt_I": "04", "dt_II": "4", "dt_p": "PM", "dt_MM": "22", "dt_SS": "29", "dt_f": "000000", "dt_z": "", "dt_Z": "", "dt_j": "060", "dt_jj": "60", "dt_U": "09", "dt_W": "09", "dt_c": "Mon Mar  1 16:22:29 2010", "dt_x": "03/01/10", "dt_X": "16:22:29"}
*** estimating xml filename...
xml fname is like: 10MAR01162229-M1BS-*_01_P009.XML
*** searching for xml file...
found file: /srv/imars-objects/rookery/Processed/wv_ortho_xml/10MAR01162229-M1BS-505387747080_01_P009.XML
*** extracting properties from .xml...
 -p IMD_NUMROWS=8192  -p IMD_NUMCOLUMNS=9216  -p ABSCALFACTOR_BAND_C=0.00909474  -p ABSCALFACTOR_BAND_B=0.01257455  -p ABSCALFACTOR_BAND_G=0.00963636  -p ABSCALFACTOR_BAND_Y=0.00501895  -p ABSCALFACTOR_BAND_R=0.01098462  -p ABSCALFACTOR_BAND_RE=0.00447579  -p ABSCALFACTOR_BAND_N=0.01217436  -p ABSCALFACTOR_BAND_N2=0.00888421  -p FIRSTLINETIME=2010-03-01_16:22:29.561650  -p MEANSUNEL=51.7  -p MEANSUNAZ=148.2  -p MEANSATEL=63.3  -p MEANSATAZ=351.4  -p MEANOFFNADIRVIEWANGLE=23.8  -p CLOUDCOVER=0.0  -p MEANINTRACKVIEWANGLE=22.5  -p MEANCROSSTRACKVIEWANGLE=-8.0  -p SATID=WV02  -p MODE=FullSwath  -p SCANDIRECTION=Forward  -p FILENAME=10MAR01162229-M1BS-505387747080_01_P009.NTF 
*** formatting ts for gee...
2010-03-01T16:22:29
*** transferring image and metadata...
Started upload task with ID: 7HQTMZY52VE2K6E4ZIOX3TEX
gee-uploads/gbucket_to_gee_w_metadata_rookery.sh: line 103: -p: command not found
done!
[...]
20200929T162715_03_P007_WV03_ClassMap_vB2_Rookery_wDEM ***
*** parsing metadata...
{"dt_Y": "2020", "dt_m": "09", "dt_d": "29", "dt_H": "16", "dt_M": "27", "dt_S": "15", "number": "03", "pass_n": "007", "sat_n": "03", "adjustments_version": "vB2", "dt_a": "Tue", "dt_A": "Tuesday", "dt_w": "2", "dt_dd": "29", "dt_b": "Sep", "dt_B": "September", "dt_mm": "9", "dt_y": "20", "dt_HH": "16", "dt_I": "04", "dt_II": "4", "dt_p": "PM", "dt_MM": "27", "dt_SS": "15", "dt_f": "000000", "dt_z": "", "dt_Z": "", "dt_j": "273", "dt_jj": "273", "dt_U": "39", "dt_W": "39", "dt_c": "Tue Sep 29 16:27:15 2020", "dt_x": "09/29/20", "dt_X": "16:27:15"}
*** estimating xml filename...
xml fname is like: 20SEP29162715-M1BS-*_03_P007.XML
*** searching for xml file...
found file: /srv/imars-objects/rookery/Processed/wv_ortho_xml/20SEP29162715-M1BS-504649660010_03_P007.XML
*** extracting properties from .xml...
 -p IMD_NUMROWS=9911  -p IMD_NUMCOLUMNS=10651  -p ABSCALFACTOR_BAND_C=0.01397474  -p ABSCALFACTOR_BAND_B=0.01772364  -p ABSCALFACTOR_BAND_G=0.01316364  -p ABSCALFACTOR_BAND_Y=0.00672  -p ABSCALFACTOR_BAND_R=0.01020364  -p ABSCALFACTOR_BAND_RE=0.00606316  -p ABSCALFACTOR_BAND_N=0.01170909  -p ABSCALFACTOR_BAND_N2=0.01034947  -p FIRSTLINETIME=2020-09-29_16:27:15.768850  -p MEANSUNEL=59.1  -p MEANSUNAZ=155.4  -p MEANSATEL=51.7  -p MEANSATAZ=236.3  -p MEANOFFNADIRVIEWANGLE=34.3  -p CLOUDCOVER=0.452  -p MEANINTRACKVIEWANGLE=-22.8  -p MEANCROSSTRACKVIEWANGLE=-26.4  -p SATID=WV03  -p MODE=FullSwath  -p SCANDIRECTION=Forward  -p FILENAME=20SEP29162715-M1BS-504649660010_03_P007.NTF 
*** formatting ts for gee...
2020-09-29T16:27:15
*** transferring image and metadata...
Started upload task with ID: LLLTVKJSFK7XLNB3OI5C4LSS
gee-uploads/gbucket_to_gee_w_metadata_rookery.sh: line 103: -p: command not found
done!

Full log retained in the repo on manglillo. The line 103: -p: command not found issue is a bug I have now fixed. The following metadata was left off due to this bug:

        -p country=${country} \
        -p generator=${generator} \
        -p classifier=${classifier}

We are not using this metadata anyway, so I do not see a need to re-run the script.

USF-IMARS / wv-land-cover

update rookery imagery #46

steps: