big pgc batches ingest

[7yl4r]

Using this issue to track progress on the ingest of WV images from PGC.

• [x] 11 batches downloaded
• [x] batches rsynced to tpa_pgs
• [x] all $batch/imagery/_$batch/ dirs merged into pgc_batches/imagery on thing2
• [x] merged batches available on tpa_pgs
• files divided up into directories within ./migration/ [ref] (regions based on this)
  • [x] alab
  • [x] cetx
  • [x] east la
  • [x] fl bb (big_bend)
  • [x] fl pan
  • [x] miss
  • [x] monroe
  • [x] ne fl
  • [x] ne tx
  • [ ] ne tx wv3
  • [x] se fl (fl_se)
  • [x] sw fl (fl_sw)
  • [ ] sw tx 2
  • [x] sw tx
  • [x] west la
• [ ] files ingested into imars_product_metadata & imars-objects w/ imars-etl

[7yl4r]

For reference there are 3799 directories total, mostly wv02 and a few wv03 and wv01.

[root@thing2 imars-objects]# ls pgc_batches/imagery/ | wc -l
3799

[root@thing2 imars-objects]# find pgc_batches/imagery/ -maxdepth 1 -name WV01* | wc -l
5
[root@thing2 imars-objects]# find pgc_batches/imagery/ -maxdepth 1 -name WV02* | wc -l
3174
[root@thing2 imars-objects]# find pgc_batches/imagery/ -maxdepth 1 -name WV03* | wc -l
620

[7yl4r]

A few files somehow ended up without subdirectories... so let's fix that real quick:

[root@thing2 imars-objects]# find pgc_batches/imagery/ -maxdepth 1 -type f | wc -l
35

fixed with fix_dirless_files.sh.

Updated directory count:

[root@userproc imagery]# ls | wc -l
3766

Counts of actual M1BS xml files by satellite:

[root@thing2 imagery]# find ./ -type f -name WV02*M1BS*xml  | wc -l
9565
[root@thing2 imagery]# find ./ -type f -name WV03*M1BS*xml  | wc -l
1883

Looks like the WV01 files are all P1BS. This gives us a total of 11448 potentially usable files.

[7yl4r]

I'm finally ready to check off this last box.

@mjm8 or @cestes-19 : can you help me make lists of files based on the "region"s we decided on? I need a list of files within each region for each region to ingest them.

I know we've talked about this before but I guess I should have written it down. Alternatively, if you can move or copy files into subdirectories for each region (like I think we did for texas_se & texas_ne), that works too.

[mjm8]

I'm planning on doing this for each region as I get to them for processing. I'll try to put them in the FilesToMove directory if I have access.

On Tue, Nov 26, 2019 at 1:30 PM Tylar notifications@github.com wrote:

I'm finally ready to check off this last box.

@mjm8 https://github.com/mjm8 or @cestes-19 https://github.com/cestes-19 : can you help me make lists of files based on the "region"s we decided on? I need a list of files within each region for each region to ingest them.

I know we've talked about this before but I guess I should have written it down. Alternatively, if you can move or copy files into subdirectories for each region (like I think we did for texas_se & texas_ne), that works too.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/22?email_source=notifications&email_token=AH7I4KGMSXOKVH4C2DYK23DQVWBNPA5CNFSM4JFHSWFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFHK56I#issuecomment-558804729, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7I4KE4ZJ42U7TVXLUCRPTQVWBNPANCNFSM4JFHSWFA .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South, KRC 3117 St Petersburg, FL 33701-5016 727-553-1186

[7yl4r]

copying to IMaRS systems:

• [x] alab
• [x] big_bend
• [x] c_texas
• [x] east_la
• [x] fl_ne
• [x] fl_se
• [x] fl_sw (west_fl_pen)
• [x] ne_texas
• [x] miss
• [X] monroe
• [x] panhandle
• [x] sw_texas
• [x] west_la

[7yl4r]

@mjm8 what are the ne tx wv3 and sw tx 2 folders? Should combine the contents with tx_ne and tx_sw, respectively?

[mjm8]

Tylar, Which directories are you referring to?

On Mon, Feb 24, 2020 at 10:22 AM Tylar notifications@github.com wrote:

@mjm8 https://github.com/mjm8 what are the ne tx wv3 and sw tx 2 folders? Should combine the contents with tx_ne and tx_sw, respectively?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/22?email_source=notifications&email_token=AH7I4KFNN6UWCXXOGMQVWW3REPQ3XA5CNFSM4JFHSWFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMYG4BQ#issuecomment-590376454, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7I4KELT6YEB6LZ5KCK3RLREPQ3XANCNFSM4JFHSWFA .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South, KRC 3117 St Petersburg, FL 33701-5016 727-553-1186 [image: USF Bull Logo]

[7yl4r]

Oh, I don't see them anymore. Used to be in /srv/imars-objects/tpa_pgs/ I think, but since I don't see them we've got no problem. Nevermind.

[mjm8]

Okay no worries!

On Mon, Feb 24, 2020 at 2:38 PM Tylar notifications@github.com wrote:

Oh, I don't see them anymore. Used to be in /srv/imars-objects/tpa_pgs/ I think, but since I don't see them we've got no problem. Nevermind.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/22?email_source=notifications&email_token=AH7I4KFIIQP35J5JJ7LCWFDREQO3FA5CNFSM4JFHSWFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMZH32Q#issuecomment-590511594, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7I4KHCX6W5NQRNEYW324TREQO3FANCNFSM4JFHSWFA .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South, KRC 3117 St Petersburg, FL 33701-5016 727-553-1186 [image: USF Bull Logo]

[7yl4r]

loading WV02 M1bs ntf & xml files into IMaRS product database:

• [x] alab (alabama)
• [x] big_bend
• [x] c_texas
• [x] east_la
• [x] fl_ne
• [x] fl_se
• [x] fl_sw (west_fl_pen)
• [ ] ne_texas (texas_ne)
• [x] miss
• [x] monroe
• [x] panhandle
• [x] sw_texas (texas_sw)
• [x] west_la

[7yl4r]

@mjm8 : the ne_texas directory has only worldview 3 files. Is that expected or did we lose something somewhere?

[mjm8]

Hmm not sure what happened there. I have all of them on Circe. Should I transfer a copy of the WV2 files back?

On Wed, Mar 11, 2020 at 12:17 PM Tylar notifications@github.com wrote:

@mjm8 https://github.com/mjm8 : the ne_texas directory has only worldview 3 files. Is that expected or did we lose something somewhere?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/22#issuecomment-597727775, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7I4KEIMM5YBD3T2AREZ2TRG62RRANCNFSM4JFHSWFA .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South, KRC 3117 St Petersburg, FL 33701-5016 727-553-1186 [image: USF Bull Logo]

[7yl4r]

Yes, that's probably the easiest way to restore them.

On Wed, Mar 11, 2020 at 2:59 PM mjm8 notifications@github.com wrote:

Hmm not sure what happened there. I have all of them on Circe. Should I transfer a copy of the WV2 files back?

On Wed, Mar 11, 2020 at 12:17 PM Tylar notifications@github.com wrote:

@mjm8 https://github.com/mjm8 : the ne_texas directory has only worldview 3 files. Is that expected or did we lose something somewhere?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/USF-IMARS/wv2-processing/issues/22#issuecomment-597727775 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AH7I4KEIMM5YBD3T2AREZ2TRG62RRANCNFSM4JFHSWFA

.

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South, KRC 3117 St Petersburg, FL 33701-5016 727-553-1186 [image: USF Bull Logo]

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/22#issuecomment-597811263, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIAV7X57T6FAB2CY4GUEN3RG7NSLANCNFSM4JFHSWFA .

--

===========================================

Tylar Murray http://tylar.info, Ph.D.

IMaRS http://imars.marine.usf.edu/ Research Systems & Software Engineer

USF CMS http://marine.usf.edu - KRC 3119-B

------------------------------------------

schedule : g-calendar

https://calendar.google.com/calendar?cid=NWRuOHRubTBmczlmZjN0cTVhMGczbnBqbXNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ, youcanbookme https://7yl4r.youcanbook.me/

===========================================

[mjm8]

Okay, copying now.

On Wed, Mar 11, 2020 at 3:02 PM Tylar notifications@github.com wrote:

Yes, that's probably the easiest way to restore them.

On Wed, Mar 11, 2020 at 2:59 PM mjm8 notifications@github.com wrote:

Hmm not sure what happened there. I have all of them on Circe. Should I transfer a copy of the WV2 files back?

On Wed, Mar 11, 2020 at 12:17 PM Tylar notifications@github.com wrote:

@mjm8 https://github.com/mjm8 : the ne_texas directory has only worldview 3 files. Is that expected or did we lose something somewhere?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <

https://github.com/USF-IMARS/wv2-processing/issues/22#issuecomment-597727775

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AH7I4KEIMM5YBD3T2AREZ2TRG62RRANCNFSM4JFHSWFA

.

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South, KRC 3117 St Petersburg, FL 33701-5016 727-553-1186 [image: USF Bull Logo]

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub < https://github.com/USF-IMARS/wv2-processing/issues/22#issuecomment-597811263 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAIAV7X57T6FAB2CY4GUEN3RG7NSLANCNFSM4JFHSWFA

.

--

===========================================

Tylar Murray http://tylar.info, Ph.D.

IMaRS http://imars.marine.usf.edu/ Research Systems & Software

Engineer

USF CMS http://marine.usf.edu - KRC 3119-B

------------------------------------------

schedule : g-calendar

< https://calendar.google.com/calendar?cid=NWRuOHRubTBmczlmZjN0cTVhMGczbnBqbXNAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ

, youcanbookme https://7yl4r.youcanbook.me/

===========================================

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/wv2-processing/issues/22#issuecomment-597812516, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7I4KCFBTNLLJMAYWKNCQTRG7N3ZANCNFSM4JFHSWFA .

-- Matt McCarthy, Ph.D. Biological Oceanography College of Marine Science University of South Florida 140 7th Avenue South, KRC 3117 St Petersburg, FL 33701-5016 727-553-1186 [image: USF Bull Logo]