upload v3 rookery classmaps to GEE

[7yl4r]

from @tisayr :

In the link below are the classification maps for Rookery (reprocessed orthorectification with DEM).

\datashare\regions\rookery\Processed\RookeryFinal\ReprocessedWithDEM\FirstClassificVersion\

Could you please upload to GEE the *ClassificMapfullClass Rookery-wDEM_v3_DEM.tif?

If you need the reprocessed full band Rrs images for your objective, those are the *Rrs Rookery-wDEM_v3_DEM.tif

Let me know when they classMaps are available in GEE.

[7yl4r]

1. [x] deleted old images in rookery bucket
2. [x] uploaded new files using gsutil
3. [ ] create GEE collection using gbucket_to_gee_w_metadata.sh
4. [ ] update GEE data access module(s) in GEE script editor

(2):

(base) tylar@manglilloo:/srv/imars-objects/rookery/Processed/RookeryFinal/ReprocessedWithDEM/FirstClassificVersion$ gsutil cp *ClassificMap_fullClass_\ Rookery-wDEM_v3_DEM.tif gs://rookery-wv-classmaps | tee ~/wv_classMaps_upload_rookery_2023-02-28.log

(3):

./gbucket_to_gee_w_metadata_rookery.sh rookery-wv-classmaps /srv/imars-objects/rookery/Processed/wv_classMaps_rgb users/tylarmurray/nerrs_rookery_v02 

These spaces in the filenames are giving me trouble. I tried renaming the files but ran into more trouble:

(base) tylar@manglilloo:/srv/imars-objects/rookery/Processed/RookeryFinal/ReprocessedWithDEM/FirstClassificVersion$ sudo find . -name "* *" -type f | rename 's/ //g'
find: ‘.’: Permission denied
find: Failed to restore initial working directory: /srv/imars-objects/rookery/Processed/RookeryFinal/ReprocessedWithDEM/FirstClassificVersion: Permission denied
(base) tylar@manglilloo:/srv/imars-objects/rookery/Processed/RookeryFinal/ReprocessedWithDEM/FirstClassificVersion$ find . -name "* *" -type f | rename 's/ //g'
[...]
Can't rename ./20200908T155700_03_P002_WV03_RGB_ Rookery-wDEM.tif ./20200908T155700_03_P002_WV03_RGB_Rookery-wDEM.tif: Permission denied
Can't rename ./20170622T161950_01_P011_WV02_RGB_ Rookery-wDEM_browse.png ./20170622T161950_01_P011_WV02_RGB_Rookery-wDEM_browse.png: Permission denied
Can't rename ./20180318T162353_01_P012_WV02_RGB_ Rookery-wDEM.tif ./20180318T162353_01_P012_WV02_RGB_Rookery-wDEM.tif: Permission denied
Can't rename ./20200908T155659_01_P001_WV03_ClassificMap_fullClass_ Rookery-wDEM_v3_DEM.tif ./20200908T155659_01_P001_WV03_ClassificMap_fullClass_Rookery-wDEM_v3_DEM.tif: Permission denied
Can't rename ./20160325T161214_01_P008_WV03_ClassificMap_fullClass_ Rookery-wDEM_v3_DEM.tif ./20160325T161214_01_P008_WV03_ClassificMap_fullClass_Rookery-wDEM_v3_DEM.tif: Permission denied
Can't rename ./20130321T163448_01_P001_WV02_Rrs_ Rookery-wDEM_v3_DEM.tif ./20130321T163448_01_P001_WV02_Rrs_Rookery-wDEM_v3_DEM.tif: Permission denied
Can't rename ./20190513T162926_01_P003_WV03_Rrs_ Rookery-wDEM_v3_DEM.tif ./20190513T162926_01_P003_WV03_Rrs_Rookery-wDEM_v3_DEM.tif: Permission denied

[7yl4r]

NOTE: the spaces in the filenames were removed and this process restarted; I have attempted to update the commands below.

1. [x] deleted old images in rookery bucket
2. [x] uploaded new files using gsutil
3. [x] create GEE collection using gbucket_to_gee_w_metadata.sh
4. [x] gee.ImageCollection set to "anyone can read" using code.earthengine.google.com
5. [x] updated GEE data access module(s) using code.earthengine.google.com:
   • users/tylarmurray/nerrs/classmap_helpers.habitat_rookery value changed from nerrs_rookery to nerrs_rookery_v02

(2):

(base) tylar@manglilloo:/srv/imars-objects/rookery/Processed/RookeryFinal/ReprocessedWithDEM/FirstClassificVersion$ gsutil cp *ClassificMap_fullClass_Rookery-wDEM_v3_DEM.tif gs://rookery-wv-classmaps | tee ~/wv_classMaps_upload_rookery_2023-02-28.log

(3):

(base) tylar@manglilloo:~/wv-land-cover/gee-uploads$ bash ./gbucket_to_gee_w_metadata_rookery.sh rookery-wv-classmaps /srv/imars-objects/rookery/Processed/wv_ortho_xml/ users/tylarmurray/nerrs_rookery_v02 

*** Transfering file  20100301T162229_01_P009_WV02_ClassificMap_fullClass_Rookery-wDEM_v3_DEM ***
*** parsing metadata...
{"dt_Y": "2010", "dt_m": "03", "dt_d": "01", "dt_H": "16", "dt_M": "22", "dt_S": "29", "number": "01", "pass_n": "009", "sat_n": "02", "dt_a": "Mon", "dt_A": "Monday", "dt_w": "1", "dt_dd": "1", "dt_b": "Mar", "dt_B": "March", "dt_mm": "3", "dt_y": "10", "dt_HH": "16", "dt_I": "04", "dt_II": "4", "dt_p": "PM", "dt_MM": "22", "dt_SS": "29", "dt_f": "000000", "dt_z": "", "dt_Z": "", "dt_j": "060", "dt_jj": "60", "dt_U": "09", "dt_W": "09", "dt_c": "Mon Mar  1 16:22:29 2010", "dt_x": "03/01/10", "dt_X": "16:22:29"}
*** estimating xml filename...
xml fname is like: 10MAR01162229-M1BS-*_01_P009.XML
*** searching for xml file...
found file: /srv/imars-objects/rookery/Processed/wv_ortho_xml/10MAR01162229-M1BS-505387747080_01_P009.XML
*** extracting properties from .xml...
python3: can't open file './wv_classify/read_wv_xml.py': [Errno 2] No such file or directory

*** formatting ts for gee...
2010-03-01T16:22:29
*** transferring image and metadata...
Started upload task with ID: 33EESCBJGFFMUDNK56RGRPHI
done!

[...]

*** Transfering file  20200929T162717_03_P008_WV03_ClassificMap_fullClass_Rookery-wDEM_v3_DEM ***
*** parsing metadata...
{"dt_Y": "2020", "dt_m": "09", "dt_d": "29", "dt_H": "16", "dt_M": "27", "dt_S": "17", "number": "03", "pass_n": "008", "sat_n": "03", "dt_a": "Tue", "dt_A": "Tuesday", "dt_w": "2", "dt_dd": "29", "dt_b": "Sep", "dt_B": "September", "dt_mm": "9", "dt_y": "20", "dt_HH": "16", "dt_I": "04", "dt_II": "4", "dt_p": "PM", "dt_MM": "27", "dt_SS": "17", "dt_f": "000000", "dt_z": "", "dt_Z": "", "dt_j": "273", "dt_jj": "273", "dt_U": "39", "dt_W": "39", "dt_c": "Tue Sep 29 16:27:17 2020", "dt_x": "09/29/20", "dt_X": "16:27:17"}
*** estimating xml filename...
xml fname is like: 20SEP29162717-M1BS-*_03_P008.XML
*** searching for xml file...
found file: /srv/imars-objects/rookery/Processed/wv_ortho_xml/20SEP29162717-M1BS-504649660010_03_P008.XML
*** extracting properties from .xml...
python3: can't open file './wv_classify/read_wv_xml.py': [Errno 2] No such file or directory

*** formatting ts for gee...
2020-09-29T16:27:17
*** transferring image and metadata...
Started upload task with ID: 4RBOKT6KDTURHPBKB4GYUXVX
done!

The no such file on read_wv_xml.py is not a good sign. This probably means that a lot of metadata was not uploaded to GEE.

The fix for this issue would be to delete the asset in code.earthengine.google.com and redo step (3) after fixing the can't find file issue.

[7yl4r]

The issue with (3) above was because gbucket_to_gee_w_metadata_rookery expects to be run from the directory above; ie:

(base) tylar@manglilloo:~/wv-land-cover/$ bash ./gee_uploads/gbucket_to_gee_w_metadata_rookery.sh rookery-wv-classmaps /srv/imars-objects/rookery/Processed/wv_ortho_xml/ users/tylarmurray/nerrs_rookery_v02 

[7yl4r]

I was able to run (3) above without deleting the collection (because I couldn't figure out how to delete the collection). It seems to be updating the images with the new information.

All looks good. :clinking_glasses: