search

NASA-PDS / operations

Tickets for the PDSEN Operations Team
Other
5 stars 1 forks source link

[nssdca-delivery] urn:nasa:pds:gbo.ast.catalina.survey::1.0 20220520 #329

Open beatricemueller opened 1 year ago

beatricemueller commented 1 year ago

Discipline Node Information

NOTE: If you have multiple delivery packages, we strongly encourage you to submit these in batches of 3 to 10 per issue with one ZIP file of the packages and another ZIP file of the validation reports. Please use a descriptive title, such as "Node Mission misc batch #".

Engineering Node Process

See the internal EN process at https://pds-engineering.jpl.nasa.gov/content/nssdca_interface_process

beatricemueller commented 1 year ago

@c-suh Note that the validation report might have errors, but any file that throws an error is pulled and is not submitted and will be submitted subsequently once fixed. This is an acceptable approach as discussed in #252

smclaughlin7 commented 1 year ago

@beatricemueller Like https://github.com/NASA-PDS/operations/issues/327, this CSS SIP has an old style SIP manifest file, where many versions of each collection are included. This SIP will likely take over a month to ingest and archive to tape.

Could you please regenerate this CSS SIP with a new style where only the latest version of each collection (for that delivery) is included in the SIP manifest file? For exampe, the previos CSS SIP, https://github.com/NASA-PDS/operations/issues/317 (nssdurn:nasa:pds:gbo.ast.catalina.survey::1.0 20220506) was the new style. It took us about a week to ingest and fully archive to tape. Going forward all CSS SIP should be in the new style.

@c-suh Could you hold off on processing this packing until this is resolved? Thanks!

beatricemueller commented 1 year ago

@c-suh @smclaughlin7 Corrected delivery package is now available https://sbnarchive.psi.edu/pds4/surveys/catalina_extras/sips/deltas/20220520_deltas.zip Note that the validation report has not changed.

smclaughlin7 commented 1 year ago

@beatricemueller Thanks to you and Jesse for making a corrected delivery package so quickly! The corrected SIP manifest file only has the last version of each collection product, as expected for the new style CSS SIPs.

beatricemueller commented 1 year ago

Stef and Catherine,

Jesse, made sips for all the rest of CSS for 2022. There are about 33 packages. How would you like me to put them into gitub. One at the time? or as batches of 10? After submitting to github, wait until they are pre-ingested or ingested at NSSDC before the next package or batch is submitted? Let me know what would be the most convenient or easiest for you.

Thanks,

Bea

On Jan 10, 2023, at 12:50 PM, smclaughlin7 @.***> wrote:

@beatricemueller https://www.google.com/url?q=https://github.com/beatricemueller&source=gmail-imap&ust=1673985032000000&usg=AOvVaw10KvpsvQNLvs7_hLSVdjss Thanks to you and Jesse for making a corrected delivery package so quickly! The corrected SIP manifest file only has the last version of each collection product, as expected for the new style CSS SIPs.

— Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/329%23issuecomment-1377768461&source=gmail-imap&ust=1673985032000000&usg=AOvVaw0ykpsMpmjDp1z9r2lV5Men, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AKLIINGH77TYWL56AZXTERDWRW4QNANCNFSM6AAAAAATVZRQTE&source=gmail-imap&ust=1673985032000000&usg=AOvVaw0GhyKWRPBukLMvXfGaxA4F. You are receiving this because you were mentioned.

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

c-suh commented 1 year ago

@beatricemueller a quick check-in on our understanding of a "package" to be a single set of 5 files (2 .xml and 3 .tab)? For example, the corrected delivery package posted a few comments above had 2 sets of files, or 2 packages. If so, yes - batches of up to 11 packages (to make 3 submissions of 11) would be fine. However, @smclaughlin7 might have a preference for smaller batches and I will defer to her on this matter, as well as on the pacing of the submissions. So, Stef, let us know!

These 2 corrected sets have been posted for NSSDCA processing. From tomorrow, you can check the status at https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp using the SIP LID below:

SIP LID: urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220520_delta_20230109175434264653 urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220520_delta_20230110154540998409

smclaughlin7 commented 1 year ago

@c-suh @beatricemueller This Catalina Sky Survey SIP is an old style 'delta' where the manifest file contains many versions of each collection product, therefore the NSSDCA will not ingest and archive it:

SIP LID: urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220520_delta_20230109175434264653

We will ingest and archvie this CSS SIP which is in the new style where the manifest file contains only the lastest version of each collection for that delivery:

SIP LID: urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220520_delta_20230110154540998409

beatricemueller commented 1 year ago

To be explicit: I meant 33 submissions with ..../sips/deltas/yyyymmdd_deltas.zip & ....validation/yyyymmdd.tar.gz

smclaughlin7 commented 1 year ago

I discussed these 33 CSS SIPs with our NSSDCA ingest team, and they prefer receiving 5 CSS SIPs at one time. Since it will take us about 3 week to ingest -- download and register all the products in our ingest database -- a batch of 5 SIPs, we propose that @beatricemueller submits a batch of 5 CSS SIPs every 3 weeks to Engineering Node/GitHub for @c-suh to check and post for the NSSDCA to download and process. Is this acceptable?

beatricemueller commented 1 year ago

@c-suh @smclaughlin7 This one had the same problem as both zip files (old and corrected were included). Here is now the correct one: https://sbnarchive.psi.edu/pds4/surveys/catalina_extras/sips/deltas/20220520_deltas.zip

beatricemueller commented 1 year ago

This will work fine for me. As we already have 20220513 and 20220520 in the works, I will submit the next 5 in 3 weeks. Do you want them bundled into 1 submission or 5 separate submissions on github?

Bea

On Jan 11, 2023, at 10:44 AM, smclaughlin7 @.***> wrote:

I discussed these 33 CSS SIPs with our NSSDCA ingest team, and they prefer receiving 5 CSS SIPs at one time. Since it will take us about 3 week to ingest -- download and register all the products in our ingest database -- a batch of 5 SIPs, we propose that @beatricemueller https://www.google.com/url?q=https://github.com/beatricemueller&source=gmail-imap&ust=1674063851000000&usg=AOvVaw1X5x8iW7oAwqUkeWrpihDD submits a batch of 5 CSS SIPs every 3 weeks to Engineering Node/GitHub for @c-suh https://www.google.com/url?q=https://github.com/c-suh&source=gmail-imap&ust=1674063851000000&usg=AOvVaw3mAoHg39eVcwuAnMdPL612 to check and post for the NSSDCA to download and process. Is this acceptable?

— Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/329%23issuecomment-1379257552&source=gmail-imap&ust=1674063851000000&usg=AOvVaw0b0KEPpwRrcbZmHK_ofYJ1, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AKLIINGCYUBD7P4D5M3ASJDWR3WOVANCNFSM6AAAAAATVZRQTE&source=gmail-imap&ust=1674063851000000&usg=AOvVaw2qETBBV14CqbRX8ow37OXl. You are receiving this because you were mentioned.

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

smclaughlin7 commented 1 year ago

@beatricemueller A heads up about SIP LID:

urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220520_delta_20230110154540998409

Our front-end process flagged these two collection products in the SIP manifest file (https://pds.nasa.gov/data/pds4/manifests/2022/gbo.ast.catalina.survey_v1.0_20220520_sip_delta_20230110154540998409_v1.0.tab) as invalid because the checksum of the downloaded manifest file did not match the checksum in manifest:

urn:nasa:pds:gbo.ast.catalina.survey:data_partially_processed::181.0 urn:nasa:pds:gbo.ast.catalina.survey:data_raw::181.0

Since the checksums do not match we will not ingest and archive those two collection products but we will ingest and archive the associated basic products. This means that we would be able to return, if necessary, all the basic products associated with those two collection versions but not the actual collection products.

Perhaps Jesse may want to check into this for future CSS SIPs?

Thanks!

smclaughlin7 commented 1 year ago

Do you want them bundled into 1 submission or 5 separate submissions on github?

@c-suh has the final say, but 1 Github submission that contains 5 CSS Deep Archive/SIP packages is OK with me/NSSDCA.

beatricemueller commented 1 year ago

Does this pertain to the most recent corrected version I posted a few minutes ago or the version posted yesterday?

Beatrice

On Jan 11, 2023, at 10:55 AM, smclaughlin7 @.***> wrote:

@beatricemueller https://www.google.com/url?q=https://github.com/beatricemueller&source=gmail-imap&ust=1674064545000000&usg=AOvVaw17uPMrMNNRZcVoZKbJlhj_ A heads up about SIP LID:

urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220520_delta_20230110154540998409

Our front-end process flagged these two collection products in the SIP manifest file (https://pds.nasa.gov/data/pds4/manifests/2022/gbo.ast.catalina.survey_v1.0_20220520_sip_delta_20230110154540998409_v1.0.tab https://www.google.com/url?q=https://pds.nasa.gov/data/pds4/manifests/2022/gbo.ast.catalina.survey_v1.0_20220520_sip_delta_20230110154540998409_v1.0.tab&source=gmail-imap&ust=1674064545000000&usg=AOvVaw1Q2vSBmQNtEl3c04KM4ADy) as invalid because the checksum of the downloaded manifest file did not match the checksum in manifest:

urn:nasa:pds:gbo.ast.catalina.survey:data_partially_processed::181.0 urn:nasa:pds:gbo.ast.catalina.survey:data_raw::181.0

Since the checksums do not match we will not ingest and archive those two collection products but we will ingest and archive the associated basic products. This means that we would be able to return, if necessary, all the basic products associated with those two collection versions but not the actual collection products.

Perhaps Jesse may want to check into this for future CSS SIPs?

Thanks!

— Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/329%23issuecomment-1379270780&source=gmail-imap&ust=1674064545000000&usg=AOvVaw1s_035udVBqD47ypVnr-Ec, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AKLIINF7VIYB7AB2EY4P4WDWR3XZ7ANCNFSM6AAAAAATVZRQTE&source=gmail-imap&ust=1674064545000000&usg=AOvVaw3_gFgijKkDQK_d8GI_YNr2. You are receiving this because you were mentioned.

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

smclaughlin7 commented 1 year ago

@c-suh @smclaughlin7 This one had the same problem as both zip files (old and corrected were included). Here is now the correct one: https://sbnarchive.psi.edu/pds4/surveys/catalina_extras/sips/deltas/20220520_deltas.zip

@beatricemueller Oops! I wasn't clear. @c-suh had posted the contents of the zip file for 2022-05-02 that you sent yesterday for the NSSDCA to download. That zip file indeed contained both the old style and correct (new style) 'delta' SIPs. Our front-end has already auto-downloaded those two SIPs. However we will only ingest and archive the correct (new style) 'delta' SIP LIDVID:

urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220520_delta_20230110154540998409

And we will manually flag the old style 'delta' SIP LID as Failed Ingest:

SIP LID: urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220520_delta_20230109175434264653

@c-suh No need to repost the SIPs for this Github submission.

smclaughlin7 commented 1 year ago

Does this pertain to the most recent corrected version I posted a few minutes ago or the version posted yesterday?

This pertains to the most recent corrected version you posted a few minutes (~30) ago. I just want to make sure that @c-suh does not post this redundant corrected version of the SIP for us/NSSDCA to download. Thanks!

c-suh commented 1 year ago

@smclaughlin7 you preemptively answered my question; thanks!

@beatricemueller in case you don't notice the "thumbs up" I gave to Stef's comment (because the icon is quite small and probably doesn't trigger any notifications for anyone), 1 GitHub submission for 5 packages sounds good! And as you pointed out there are already 2 packages in the works, we will expect that next submission as you proposed in about 3 weeks. Thank you!

smclaughlin7 commented 1 year ago

@beatricemueller @c-suh We/NSSDCA may be finished ingesting the two corrected CSS SIPs for 2022-05-13 and 2022-05-22 by middle of next week. Would you be willing to do 1 GitHub submission for 5 Deep Archive packages/SIPs sometime next week? Then plan for the next Github submission of 5 packages 3 weeks after that?

One question for @beatricemueller: Were the 33 CSS Deep Archive packages generated using the new style where a SIP manifest file contains only the last version of each collection product for that delivery? Thanks!

beatricemueller commented 1 year ago

Sure!

Beatrrice

On Jan 11, 2023, at 12:40 PM, smclaughlin7 @.***> wrote:

@beatricemueller https://www.google.com/url?q=https://github.com/beatricemueller&source=gmail-imap&ust=1674070811000000&usg=AOvVaw1CZsOSGfdYd3FDtjkehvbd @c-suh https://www.google.com/url?q=https://github.com/c-suh&source=gmail-imap&ust=1674070811000000&usg=AOvVaw2vdViIyP9ukqxYzNjkHgZy We/NSSDCA may be finished ingesting the two corrected CSS SIPs for 2022-05-13 and 2022-05-22 by middle of next week. Would you be willing to do 1 GitHub submission for 5 Deep Archive packages/SIPs sometime next week? Then plan for the next Github submission of 5 packages 3 weeks after that?

One question for @beatricemueller https://www.google.com/url?q=https://github.com/beatricemueller&source=gmail-imap&ust=1674070811000000&usg=AOvVaw1CZsOSGfdYd3FDtjkehvbd: Were the 33 CSS Deep Archive packages generated using the new style where a SIP manifest file contains only the last version of each collection product for that delivery? Thanks!

— Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/329%23issuecomment-1379391389&source=gmail-imap&ust=1674070811000000&usg=AOvVaw2Ti8ZiKd6D1lvtCGBnatyE, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AKLIINEQIR767LJWBS2LHYLWR4EBTANCNFSM6AAAAAATVZRQTE&source=gmail-imap&ust=1674070811000000&usg=AOvVaw0gwzoOS5uNd3Epo837AXt1. You are receiving this because you were mentioned.

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

smclaughlin7 commented 1 year ago

@beatricemueller @c-suh We/NSSDCA may be finished ingesting the two corrected CSS SIPs for 2022-05-13 and 2022-05-22 by middle of next week. Would you be willing to do 1 GitHub submission for 5 Deep Archive packages/SIPs sometime next week? Then plan for the next Github submission of 5 packages 3 weeks after that?

@beatricemueller Our Ingest team reminded me they're still ingesting a batch of SIPs from NAIF and a few SIPs from ATM. Instead of submitting the first batch of 5 CSS SIPs late next week, could you please submit it in 3 weeks as you originally proposed? I apologize for any inconvenience. Thanks! @c-suh

beatricemueller commented 1 year ago

3 weeks it is.

On Jan 12, 2023, at 6:42 AM, smclaughlin7 @.***> wrote:

@beatricemueller https://www.google.com/url?q=https://github.com/beatricemueller&source=gmail-imap&ust=1674135734000000&usg=AOvVaw0ZcSmIvmn2uz1lOWampV08 @c-suh https://www.google.com/url?q=https://github.com/c-suh&source=gmail-imap&ust=1674135734000000&usg=AOvVaw19oOj9jBYoCCGCSO-mX8JO We/NSSDCA may be finished ingesting the two corrected CSS SIPs for 2022-05-13 and 2022-05-22 by middle of next week. Would you be willing to do 1 GitHub submission for 5 Deep Archive packages/SIPs sometime next week? Then plan for the next Github submission of 5 packages 3 weeks after that?

@beatricemueller https://www.google.com/url?q=https://github.com/beatricemueller&source=gmail-imap&ust=1674135734000000&usg=AOvVaw0ZcSmIvmn2uz1lOWampV08 Our Ingest team reminded me they're still ingesting a batch of SIPs from NAIF and a few SIPs from ATM. Instead of submitting the first batch of 5 CSS SIPs late next week, could you please submit it in 3 weeks as you originally proposed? I apologize for any inconvenience. Thanks! @c-suh https://www.google.com/url?q=https://github.com/c-suh&source=gmail-imap&ust=1674135734000000&usg=AOvVaw19oOj9jBYoCCGCSO-mX8JO — Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/329%23issuecomment-1380374376&source=gmail-imap&ust=1674135734000000&usg=AOvVaw1SqYI8oUmy3nMOVV7Q3YZG, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AKLIINF3CUJDEQ4WCDYJDXLWSAC3JANCNFSM6AAAAAATVZRQTE&source=gmail-imap&ust=1674135734000000&usg=AOvVaw0s3-1SDnkk7oPsSiv49q_U. You are receiving this because you were mentioned.

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

smclaughlin7 commented 1 year ago

Great! Thanks Bea!

smclaughlin7 commented 1 year ago

@beatricemueller The NSSDCA finished ingesting this SIP (Yay!) and started archiving it products to tape:

urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220520_delta_20230110154540998409

I want to let you know that our front-end process flagged these two collection products in the SIP manifest file (https://pds.nasa.gov/data/pds4/manifests/2022/gbo.ast.catalina.survey_v1.0_20220520_sip_delta_20230110154540998409_v1.0.tab) as invalid because the checksums of the downloaded collection label and inventory files did not match the checksums in manifest:

urn:nasa:pds:gbo.ast.catalina.survey:data_partially_processed::181.0 urn:nasa:pds:gbo.ast.catalina.survey:data_raw::181.0

Since the checksums did not match we could not ingest and archive those two collection products but we did ingest and archive all associated basic products listed in the SIP manifest file. This means that we would be able to return, if necessary, all the basic products associated with those two collection versions but not the actual collection products themselves.

I suspect this is not a big issue given that most CSS collection products are frequently updated and incrementally versioned, but perhaps Jesse may want to check into this for future CSS SIPs in the 'new style'? Thanks!

beatricemueller commented 1 year ago

Hi Catherine,

I am supposed to get the next CSS Batch into github today. However, CSS Batch D (April 5) and Batch E (April 28), are still on github and not on NSSDC. As submissions from Batch C are still only Pre-Ingest on NSSDC, do you want me to hold off with submitting my batch F?

Beatrice

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

c-suh commented 1 year ago

Hi Beatrice (@beatricemueller), and thank you for checking in! Yes, we would appreciate if you were to hold off on submitting Batch F. @smclaughlin7, do you have a preference on how long to wait, e.g. 3 or 6 weeks after Batches D and E are submitted?

smclaughlin7 commented 1 year ago

@beatricemueller @c-suh Thank you both for checking in.

As submissions from Batch C are still only Pre-Ingest on NSSDC, do you want me to hold off with submitting my batch F?

The good news is we're presently ingesting Batch C (#377), SIP LIDs:

I've asked our ingest team if it's OK for Catherine to post at least CSS Batch D (#384) now so that it's in our queue. I also asked about how to long Beatrice should hold off on delivering Batch F. I should have their recommendations tomorrow. Thanks!

smclaughlin7 commented 1 year ago

@c-suh Our ingest team ask it's OK to post CSS Batches D (#384) and E (#390). @beatricemueller Could you please hold off submitted the next CSS batch to Github until the end of May.? Thanks!

beatricemueller commented 1 year ago

On May 18, 2023, at 6:07 AM, smclaughlin7 @.***> wrote:

@c-suh https://www.google.com/url?q=https://github.com/c-suh&source=gmail-imap&ust=1685020053000000&usg=AOvVaw3pzqX3VZM-OxwCvsk3yBZR Our ingest team ask it's OK to post CSS Batches D (#384 https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/384&source=gmail-imap&ust=1685020053000000&usg=AOvVaw2_Q7v29GUFfBa060Nd6Mw3) and E (#390 https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/390&source=gmail-imap&ust=1685020053000000&usg=AOvVaw3LitphUSroDfarqGmwbE35). @beatricemueller https://www.google.com/url?q=https://github.com/beatricemueller&source=gmail-imap&ust=1685020053000000&usg=AOvVaw0Q42A180B7JkUYD057r2HV Could you please hold off submitted the next CSS batch to Github until the end of May.? Thanks!

Will do.

Beatrice

— Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/329%23issuecomment-1553030527&source=gmail-imap&ust=1685020053000000&usg=AOvVaw05KgdbuYfFhTo4jMlSHruc, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AKLIINGQLO4LWQAPQ4TL5OLXGYNJHANCNFSM6AAAAAATVZRQTE&source=gmail-imap&ust=1685020053000000&usg=AOvVaw02q9C_49UwISR32ZTmcCVd. You are receiving this because you were mentioned.

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

beatricemueller commented 1 year ago

Hi everyone,

I am supposed to submit the next CSS batch (G). However, batch F I submitted end of May to github, is still there and not at NSSDCA. It also seems that most CSS at NSSDCA are still pre-ingest. Do you want me to hold off with batch G? I also have a massive OREx-OVIRS dataset ready to get submitted. Do you want me to wait on this one too?

Beatrice PS: sorry if you get this more than once as I don't know who is included in NASA-PDS/operations.

On May 18, 2023, at 06:47, Beatrice Mueller @.***> wrote:

On May 18, 2023, at 6:07 AM, smclaughlin7 @.***> wrote:

@c-suh https://www.google.com/url?q=https://github.com/c-suh&source=gmail-imap&ust=1685020053000000&usg=AOvVaw3pzqX3VZM-OxwCvsk3yBZR Our ingest team ask it's OK to post CSS Batches D (#384 https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/384&source=gmail-imap&ust=1685020053000000&usg=AOvVaw2_Q7v29GUFfBa060Nd6Mw3) and E (#390 https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/390&source=gmail-imap&ust=1685020053000000&usg=AOvVaw3LitphUSroDfarqGmwbE35). @beatricemueller https://www.google.com/url?q=https://github.com/beatricemueller&source=gmail-imap&ust=1685020053000000&usg=AOvVaw0Q42A180B7JkUYD057r2HV Could you please hold off submitted the next CSS batch to Github until the end of May.? Thanks!

Will do.

Beatrice

— Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/329%23issuecomment-1553030527&source=gmail-imap&ust=1685020053000000&usg=AOvVaw05KgdbuYfFhTo4jMlSHruc, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AKLIINGQLO4LWQAPQ4TL5OLXGYNJHANCNFSM6AAAAAATVZRQTE&source=gmail-imap&ust=1685020053000000&usg=AOvVaw02q9C_49UwISR32ZTmcCVd. You are receiving this because you were mentioned.

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

smclaughlin7 commented 1 year ago

Hi @beatricemueller, I've asked our NSSDCA ingest team if it's OK for @c-suh to cross-check and post CSS Batch F https://github.com/NASA-PDS/operations/issues/403. I should hear back from them tomorrow and will let you know then about submitting new Batch G to github. Thanks for your patience!

Regarding the massive OREx-OVIRS dataset, roughly how many records are listed manifest file and what is the data volume? My gut reaction is to submit just go ahead and submit it github.

beatricemueller commented 1 year ago

On Jun 21, 2023, at 13:31, smclaughlin7 @.***> wrote:

Hi @beatricemueller https://www.google.com/url?q=https://github.com/beatricemueller&source=gmail-imap&ust=1687984304000000&usg=AOvVaw02wnCEHXgE4uv1wu2N9D0S, I've asked our NSSDCA ingest team if it's OK for @c-suh https://www.google.com/url?q=https://github.com/c-suh&source=gmail-imap&ust=1687984304000000&usg=AOvVaw3QAlA4YH3AEqC97C2vLHBx to cross-check and post CSS Batch F #403 https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/403&source=gmail-imap&ust=1687984304000000&usg=AOvVaw1qOeJe3711TVSVZjCbZK8l. I should hear back from them tomorrow and will let you know then about submitting new Batch G to github. Thanks for your patience!

Regarding the massive OREx-OVIRS dataset, roughly how many records are listed manifest file and what is the data volume? My gut reaction is to submit just go ahead

2,342,801 records, about 1.2 TB

Bea

and submit it github.

— Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/329%23issuecomment-1601629269&source=gmail-imap&ust=1687984304000000&usg=AOvVaw336XqyvV4k6Ilrqd8ix6BN, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AKLIINB4BQ4ZQW754NLDU3LXMNK25ANCNFSM6AAAAAATVZRQTE&source=gmail-imap&ust=1687984304000000&usg=AOvVaw2UcfeHJRHekMz3bIemdYT2. You are receiving this because you were mentioned.

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

smclaughlin7 commented 1 year ago

Hi @beatricemueller, Thanks for sending the #records and #TBs for the OREx-OVIRS bundle.I'll ask our Ingest folks about submitting this big one. They might welcome a break from ingesting and archiving CSS submissions. ;-)

smclaughlin7 commented 1 year ago

Hi @beatricemueller, Our ingest folks said you should submit that big OREx-OVIRS bundle and the new CSS Batch G to EN.

Hi @c-suh, Our ingest team said it's OK to cross-check and post:

But please hold off processing and posting new CSS Batch G for now.

Thanks!

c-suh commented 1 year ago

Prematurely closed this ticket as I hastily translated "started archiving to tape" as "archived". However, upon checking these 2 SIP LIDs:

smclaughlin7 commented 1 year ago

Catherine, Thank you for keeping this ticket open for these 2 SIP LIDs:

jordanpadams commented 10 months ago

Status verified on 11/30/2023: SIPs are still awaiting ingest

jordanpadams commented 9 months ago

@beatricemueller all applicable packages have been validated by NSSDCA and submitted to Ingest. NSSDCA has now taken responsibility for an additional copy of this data.

smclaughlin7 commented 9 months ago

@beatricemueller all applicable packages have been validated by NSSDCA and submitted to Ingest. NSSDCA has now taken responsibility for an additional copy of this data.

@jordanpadams This is very nitpicky. I recommend rewording to

"...packages have been validated by NSSDCA, submitted to Ingest, and Ingested. NSSDCA has now taken responsibility..."

or perhaps

"...packages have been validated and ingested by NSSDCA. NSSDCA has now taken responsibility..."

Submitted to Ingest != Ingested. NSSDCA must first finish ingesting a SIP ("Ingested") before it can take responsibility for archiving the data. For example, it is possible for Ingest to reject a SIP. In this case, the SIP was submitted to Ingest but never ingested so the NSSDCA would not take archive responsibility.

But no worries if the original wording works for PDS purposes. Thanks!

jordanpadams commented 9 months ago

@smclaughlin7 +1 updated our procedures moving forward.

beatricemueller commented 9 months ago

I am a little confused. Some of the closed packages are still listed on the NSSDC website as pre-ingest, not ingested. (e.g #410). Has the website not been updated recently (it seemed nothing has changed for months), or did I misunderstand what it means to have an issue closed.

Bea

On Dec 20, 2023, at 11:14, smclaughlin7 @.***> wrote:

@beatricemueller https://www.google.com/url?q=https://github.com/beatricemueller&source=gmail-imap&ust=1703700857000000&usg=AOvVaw3Wew66U9rSZm-QiIo8gKTe all applicable packages have been validated by NSSDCA and submitted to Ingest. NSSDCA has now taken responsibility for an additional copy of this data.

@jordanpadams https://www.google.com/url?q=https://github.com/jordanpadams&source=gmail-imap&ust=1703700857000000&usg=AOvVaw2emJrZrq6JTspKLliNtWWX This is very nitpicky. I recommend rewording to

"...packages have been validated by NSSDCA, submitted to Ingest, and Ingested. NSSDCA has now taken responsibility..."

or perhaps

"...packages have been validated and ingested by NSSDCA. NSSDCA has now taken responsibility..."

Submitted to Ingest != Ingested. NSSDCA must first finish ingesting a SIP ("Ingested") before it can take responsibility for archiving the data. For example, it is possible for Ingest to reject a SIP. In this case, the SIP was submitted to Ingest but never ingested so the NSSDCA would not take archive responsibility.

But no worries if the original wording works for PDS purposes. Thanks!

— Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/329%23issuecomment-1864921510&source=gmail-imap&ust=1703700857000000&usg=AOvVaw0uyjnCI_auX2afr2cjOIo6, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AKLIINDYPUM5ZUFXWF3KZL3YKMTHPAVCNFSM6AAAAAATVZRQTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRUHEZDCNJRGA&source=gmail-imap&ust=1703700857000000&usg=AOvVaw31p94ndPXWKQlv22cbR7ns. You are receiving this because you were mentioned.

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

smclaughlin7 commented 9 months ago

@smclaughlin7 +1 updated our procedures moving forward. @jordanpadams updating their procedures so that an issue will be closed after it listed as "Ingested" on the NSSDCA website. They were being closed if listed as "Pre-Ingest" or "Ingested". (This was due to an oversight on my part.)

"Pre-Ingest" means the SIP has been sent to NSSDCA Ingest but is waiting to be or is being ingested. Occassionally a problem is encountered during ingest so the Github issue should remain open until the SIP has been "Ingested", which means the NSSDCA has a copy of the submitted data and accepts archive responsibility and will be writting it to tape.

jordanpadams commented 9 months ago

@beatricemueller that was my mistake getting a little over zealous. I was using the NSSDCA API to get status, and I was incorrect in my understanding of the possible statuses.

c-suh commented 8 months ago

Status: urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220520_delta_20230110154540998409 is Partial Ingest.

c-suh commented 7 months ago

Status is the same.

c-suh commented 7 months ago

Status is the same.

c-suh commented 6 months ago

urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220520_delta_20230110154540998409 is partial ingest and still in processing.

c-suh commented 6 months ago

Status is the same for this package.

c-suh commented 6 months ago

urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220520_delta_20230110154540998409 is still partial ingest at https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp.