subdividing FL for wv2 processing project

[7yl4r]

@mjm8 : I think we spoke briefly about dividing up the area we intend to cover under the microsoft grant. You may already have something better figured out, but I was thinking we could use the same subdivisions as FCMaP to divide up FL.

Based on those subdivisions I am thinking we run "big bend" and "West FL Pen." on our servers & do the others in microsoft's cloud.

[mjm8]

Hi Tylar,

Sorry for the delay. I'm back from D.C. last week, but working from home today on the NERRS and NSF grant details. I agree with the FCMaP divisions and will try to recreate those files from which to select images. I'm thinking of creating directories for each division, and subdirectories for each year within each division. Where should I place the images? Or should I go about the file collection a different way?

Best, Matt

On Mon, Jun 4, 2018 at 9:45 AM, Tylar notifications@github.com wrote:

@mjm8 https://github.com/mjm8 : I think we spoke briefly about dividing up the area we intend to cover under the microsoft grant. You may already have something better figured out, but I was thinking we could use the same subdivisions as FCMaP https://www.usgs.gov/media/images/graphic-location-map-shows-six-regions-fcmap to divide up FL.

Based on those subdivisions I am thinking we run "big bend" and "West FL Pen." on our servers & do the others in microsoft's cloud.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/3, or mute the thread https://github.com/notifications/unsubscribe-auth/Af6OKMx9K9-gokZfkOBMmELXLzdvGHLiks5t5TnmgaJpZM4UZHbN .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South St Petersburg, FL 33701-5016 727-553-1186

[7yl4r]

It is relatively trivial for me to re-organize images once I have indexed them, so we can use whatever organization paradigm works best for you (and others(?) accessing wv2 data). We can even keep using /extra_data if desired.

That being said, I would most prefer to organize the images using this imars-objects spec - which is very near to what you propose. In this case I think it would look something like:

/big_bend
    /wv2
        /...
        /2017
        /2018
/fl_se
    /wv2
        /...
        /2017
        /2018
/fl_ne
    /wv2
        /...
        /2017
        /2018
/monroe
    /wv2
        /...
        /2017
        /2018
/panhandle
    /wv2
        /...
        /2017
        /2018
/west_fl_pen
    /wv2
        /...
        /2017
        /2018

it might also be nice to split up the different files by subdividing wv2 somehow. eg:

/big_bend
    /wv2_m1bs
    /wv2_p1bs

...but only if that subdivision won't cause any issues for your (or others) processing routines.

[mjm8]

That sounds good. I'll start working on it this week.

On Mon, Jun 11, 2018 at 11:39 AM, Tylar notifications@github.com wrote:

It is relatively trivial for me to re-organize images once I have indexed them, so we can use whatever organization paradigm works best for you (and others(?) accessing wv2 data). We can even keep using /extra_data if desired.

That being said, I would most prefer to organize the images using this imars-objects spec https://github.com/USF-IMARS/IMaRS-docs/tree/master/docs/management_data/imars-objects

• which is very near to what you propose. In this case I think it would look something like:

/big_bend /wv2 /... /2017 /2018 /fl_se /wv2 /... /2017 /2018 /fl_ne /wv2 /... /2017 /2018 /monroe /wv2 /... /2017 /2018 /panhandle /wv2 /... /2017 /2018 /west_fl_pen /wv2 /... /2017 /2018

it might also be nice to split up the different files by subdividing wv2 somehow. eg:

/big_bend /wv2_m1bs /wv2_p1bs

...but only if that subdivision won't cause any issues for your (or others) processing routines.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/3#issuecomment-396287945, or mute the thread https://github.com/notifications/unsubscribe-auth/Af6OKByWg_C82zoBOZFJkB_kZTjSGx93ks5t7o8cgaJpZM4UZHbN .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South St Petersburg, FL 33701-5016 727-553-1186

[7yl4r]

The initial tests have gone well enough. Still a few bugs to iron out, but nothing major.

I would like to start processing the big bend region on our servers here. I think the easiest way to do this will be to set up ftp downloads from digital globe as we were before, using the filename convention wv2_YYYY-MM-DDThhmm_big_bend.zip, where the datetime used is the time the order is placed.

What do you think @mjm8 ?

[mjm8]

I'm glad you're ready to start processing, though we should also take advantage of the images we already have, which includes all of the Big Bend for 2014/2015. Can we start with those to test, and move to updating our database when that goes smoothly? The other thing we need to consider for Azure and NSF is that our old PGC contacts just informed me that they are no longer filling our requests for large batches of imagery. They said they're working on an alternative for us, but we may need to either find our own alternative for large scale image requests, or start downloading them individually through the DG portal. Maybe we need an intern? Enrique happens to be here right now and said he has an intern for other labs stuff who wants to get involved in remote sensing. I'll send a separate email to you and him now on that.

In the meantime, let me know if we can use the existing imagery, and where I should put that as well as how many images (2? 10? 100?)

On Wed, Jun 27, 2018 at 5:20 PM, Tylar notifications@github.com wrote:

The initial tests have gone well enough. Still a few bugs to iron out, but nothing major.

I would like to start processing the big bend region on our servers here. I think the easiest way to do this will be to set up ftp downloads from digital globe as we were before, using the filename convention wv2_YYYY-MM-DDThhmm_big_bend.zip, where the datetime used is the time the order is placed.

What do you think @mjm8 https://github.com/mjm8 ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/3#issuecomment-400832885, or mute the thread https://github.com/notifications/unsubscribe-auth/Af6OKM29kDxXdNJiHW6ZulkDPcmOdabUks5uA_ctgaJpZM4UZHbN .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South St Petersburg, FL 33701-5016 727-553-1186

[7yl4r]

how many images (2? 10? 100?)

I think we need a few 100 to keep the servers busy while I explore optimizations.

start downloading them individually through the DG portal

Ouch. If this is the case I can get creative with automating it, but it would be much better for everyone if DG exposed an API. We'll talk more on that later I suppose.

the images we already have, which includes all of the Big Bend for 2014/2015 [ ... ] and where I should put that

Oh, perfect! Ultimately we need to "load" them using this imars-etl tool I have been working on. This puts the metadata into a database and copies the file to /srv/imars-objects/big_bend/wv2/ so all the servers can find them. Do you have a list of the files or are the all in a directory?

[mjm8]

It looks like I've got 454 NTF and XML files each in one directory in Tampa. I can scp those wherever you like.

On Thu, Jun 28, 2018 at 10:47 AM, Tylar notifications@github.com wrote:

how many images (2? 10? 100?) I think we need a few 100 to keep the servers busy while I explore optimizations.

start downloading them individually through the DG portal Ouch. If this is the case I can get creative with automating it, but it would be much better for everyone if DG exposed an API. We'll talk more on that later I suppose.

the images we already have, which includes all of the Big Bend for 2014/2015 [ ... ] and where I should put that Oh, perfect! Ultimately we need to "load" them using this imars-etl https://github.com/USF-IMARS/imars-etl tool I have been working on. This puts the metadata into a database and copies the file to /srv/imars-objects/big_bend/wv2/ so all the servers can find them. Do you have a list of the files or are the all in a directory?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/3#issuecomment-401060746, or mute the thread https://github.com/notifications/unsubscribe-auth/Af6OKIvO4OQKNc1wzScO8ecWGWz2gDHmks5uBOx0gaJpZM4UZHbN .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South St Petersburg, FL 33701-5016 727-553-1186

[7yl4r]

This should work, but you might want to test it on a small number of files first. The ideal location would be mmccarthy@thing2.marine.usf.edu:/thing2/imars-objects/big_bend/wv2/2014 and .../2015

[mjm8]

My password for that directory is not working. Should it be the same as my seashell password? Or do I need to set one up?

On Thu, Jun 28, 2018 at 11:52 AM, Tylar notifications@github.com wrote:

This should work, but you might want to test it on a small number of files first. The ideal location would be mmccarthy@thing2.marine.usf. edu:/thing2/imars-objects/big_bend/wv2/2014 and .../2015

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/3#issuecomment-401083124, or mute the thread https://github.com/notifications/unsubscribe-auth/Af6OKJCWZl_HKyxdznONz38EYLB_BDHkks5uBPvZgaJpZM4UZHbN .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South St Petersburg, FL 33701-5016 727-553-1186

[7yl4r]

Should be the same. can you ssh to thing2.marine.usf.edu?

[mjm8]

Nope. Password doesn't work.

On Thu, Jun 28, 2018 at 12:03 PM, Tylar notifications@github.com wrote:

Should be the same. can you ssh to thing2.marine.usf.edu?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/3#issuecomment-401086487, or mute the thread https://github.com/notifications/unsubscribe-auth/Af6OKChW8rMilkHFLg2rhpdnuJvAunuqks5uBP5GgaJpZM4UZHbN .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South St Petersburg, FL 33701-5016 727-553-1186

[7yl4r]

Ah, okay. I guess it is not working. Noted.

Use mmccarthy@seashell.marine.usf.edu:/srv/imars-objects/big_bend/wv2/2014 then.

It will end up the in the same place, just a bit more network usage.

[mjm8]

Transferring 2014... 2015 next.

On Thu, Jun 28, 2018 at 12:43 PM, Tylar notifications@github.com wrote:

Ah, okay. I guess it is not working. Noted https://github.com/USF-IMARS/server-status/issues/104.

Use mmccarthy@seashell.marine.usf.edu:/srv/imars-objects/big_bend/wv2/2014 then.

It will end up the in the same place, just a bit more network usage.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/3#issuecomment-401099172, or mute the thread https://github.com/notifications/unsubscribe-auth/Af6OKEI3pceV9-TU-uHl3cQK2GL6wGRAks5uBQfMgaJpZM4UZHbN .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South St Petersburg, FL 33701-5016 727-553-1186

[7yl4r]

How go the transfers?

[mjm8]

I thought I transferred all the 2014 and 2015 images of Big Bend. I'll organize and send the rest of what we have when I'm back next week, but for new images we either need the intern or Frank's connections with NASA. How is the processing going?

On Tue, Jul 17, 2018, 8:30 AM Tylar notifications@github.com wrote:

How go the transfers?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/3#issuecomment-405624558, or mute the thread https://github.com/notifications/unsubscribe-auth/Af6OKGdEWXYBEiH8KlgFLn6eMdcVNK_wks5uHgL-gaJpZM4UZHbN .

[7yl4r]

I just fixed an issue that was killing 2/3 of the jobs. You can check on the running jobs in airflow here, and the output files are being saved to /srv/imars-objects/big_bend/tif_r_rs_wv2/. It looks like the permissions aren't right. I will fix that soon as well.