search

NASA-PDS / operations

Tickets for the PDSEN Operations Team
Other
5 stars 1 forks source link

[nssdca-delivery] urn:nasa:pds:gbo.ast.catalina.survey::1.0 20220513 #327

Closed beatricemueller closed 9 months ago

beatricemueller commented 1 year ago

Discipline Node Information

NOTE: If you have multiple delivery packages, we strongly encourage you to submit these in batches of 3 to 10 per issue with one ZIP file of the packages and another ZIP file of the validation reports. Please use a descriptive title, such as "Node Mission misc batch #".

Engineering Node Process

See the internal EN process at https://pds-engineering.jpl.nasa.gov/content/nssdca_interface_process

beatricemueller commented 1 year ago

@c-suh: Note that the validation report might have errors, but any file that throws an error is pulled and is not submitted and will be submitted subsequently once fixed. This is an acceptable approach as discussed in #252

c-suh commented 1 year ago

@beatricemueller noted; thank you! This set has been posted for NSSDCA processing. From tomorrow, you can check the status at https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp using the SIP LID below:

SIP LID:

smclaughlin7 commented 1 year ago

@beatricemueller This CSS SIP has an old style SIP manifest file, https://pds.nasa.gov/data/pds4/manifests/2022/gbo.ast.catalina.survey_v1.0_20220513_sip_delta_20230105190307012191_v1.0.tab, where many versions of each collection are included. This SIP will likely take over a month to ingest and archive to tape.

Could you please regenerate this CSS SIP with a new style where only the latest version of each collection (for that delivery) is included in the SIP manifest file? For exampe, the previos CSS SIP, https://github.com/NASA-PDS/operations/issues/317 (nssdurn:nasa:pds:gbo.ast.catalina.survey::1.0 20220506) was the new style. It took us about a week to ingest and fully archive to tape. Going forward all CSS SIP should be in the new style. @c-suh

beatricemueller commented 1 year ago

Jesse will regenerate. Do you want me to use the same github submission and add the corrected sip in the comments?

Bea

On Jan 10, 2023, at 8:03 AM, smclaughlin7 @.***> wrote:

@beatricemueller https://www.google.com/url?q=https://github.com/beatricemueller&source=gmail-imap&ust=1673967796000000&usg=AOvVaw3e93tzro8yotMM4Grao3SO This CSS SIP has an old style SIP manifest file, https://pds.nasa.gov/data/pds4/manifests/2022/gbo.ast.catalina.survey_v1.0_20220513_sip_delta_20230105190307012191_v1.0.tab https://www.google.com/url?q=https://pds.nasa.gov/data/pds4/manifests/2022/gbo.ast.catalina.survey_v1.0_20220513_sip_delta_20230105190307012191_v1.0.tab&source=gmail-imap&ust=1673967796000000&usg=AOvVaw38AlKhpLugPcxN5VeGKwHO, where many versions of each collection are included. This SIP will likely take over a month to ingest and archive to tape.

Could you please regenerate this CSS SIP with a new style where only the latest version of each collection (for that delivery) is included in the SIP manifest file? For exampe, the previos CSS SIP, #317 https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/317&source=gmail-imap&ust=1673967796000000&usg=AOvVaw1TaDmnjQU0pM0CVXEXfFqL (nssdurn:nasa:pds:gbo.ast.catalina.survey::1.0 20220506) was the new style. It took us about a week to ingest and fully archive to tape. Going forward all CSS SIP should be in the new style. @c-suh https://www.google.com/url?q=https://github.com/c-suh&source=gmail-imap&ust=1673967796000000&usg=AOvVaw1JBeClKCycLVdxyxZNXwg0 — Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/327%23issuecomment-1377409452&source=gmail-imap&ust=1673967796000000&usg=AOvVaw2XCWy35S6cWOyDoeD1dFLR, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AKLIINFD63FIGTRX3FOS4ALWRV23FANCNFSM6AAAAAATTQ4N54&source=gmail-imap&ust=1673967796000000&usg=AOvVaw2nT6jXUa7mYK9B-5GOQqK1. You are receiving this because you were mentioned.

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

smclaughlin7 commented 1 year ago

@beatricemueller @c-suh Bea, Thank you for working with Jesse to regenerate. I'm fine with adding the corrected SIP in the comments for this github https://github.com/NASA-PDS/operations/issues/327, but @c-suh should probably confirm.

smclaughlin7 commented 1 year ago

@beatricemueller Before Jesse regenerates, could you let him know our front-end process flagged these collection products in the SIP manifest file as invalid because the checksum of the downloaded file did not match the checksum in manifest:

urn:nasa:pds:gbo.ast.catalina.survey:data_partially_processed::160.0 thru 176.0 urn:nasa:pds:gbo.ast.catalina.survey:data_raw::160.0 thru 176.0.

Of course, the regenerated SIP manifest should only contain the last version of these collection products so the earlier versions having stale checksums should not be an issue.

Thanks!

c-suh commented 1 year ago

@beatricemueller confirming Stef's response that adding the corrected SIP in the comments of this same github submission is fine. Thank you!

smclaughlin7 commented 1 year ago

@beatricemueller Just curious. When might the corrected SIP for this 2022-05-13 delivery be submitted here to EN? Thanks! @c-suh

beatricemueller commented 1 year ago

@c-suh @smclaughlin7 oops, it slipped through the cracks: here is the corrected sip:

https://sbnarchive.psi.edu/pds4/surveys/catalina_extras/sips/deltas/20220513_deltas.zip

smclaughlin7 commented 1 year ago

@beatricemueller @c-suh That was fast! That zip file includes files the old style 'delta' SIP, e.g. gbo.ast.catalina.survey_v1.0_20220513_sip_delta_20230105190307012191_v1.0.xml. Would it be possible to make a zip file for @c-suh that contains the files for the style 'delta' SIP'? (File names with the string '20230110154527571604'). Thanks!

beatricemueller commented 1 year ago

@c-suh @smclaughlin7 ok. hopefully this has no errors: https://sbnarchive.psi.edu/pds4/surveys/catalina_extras/sips/deltas/20220513_deltas.zip

smclaughlin7 commented 1 year ago

@beatricemueller @c-suh This latest zip file looks good to me; it only includes the corrected (new style) 'delta' SIP for 2022-05-13. Thanks!

c-suh commented 1 year ago

@beatricemueller and @smclaughlin7 I've posted this most recent package! From tomorrow, you can check the status at https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp using the SIP LID below:

SIP LID:

smclaughlin7 commented 1 year ago

@beatricemueller The NSSDCA finished ingesting this SIP (Yay!) and started archiving it products to tape:

urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220513_delta_20230110154527571604

I want to let you know that our front-end process flagged these two collection products in the SIP manifest file (https://pds.nasa.gov/data/pds4/manifests/2022/gbo.ast.catalina.survey_v1.0_20220513_sip_delta_20230110154527571604_v1.0.tab) as invalid because the checksums of the downloaded collection label and inventory files did not match the checksums in manifest:

urn:nasa:pds:gbo.ast.catalina.survey:data_partially_processed::176.0 urn:nasa:pds:gbo.ast.catalina.survey:data_raw::176.0

Since the checksums did not match we could not ingest and archive those two collection products but we did ingest and archive all associated basic products listed in the SIP manifest file. This means that we would be able to return, if necessary, all the basic products associated with those two collection versions but not the actual collection products themselves.

I suspect this is not a big issue given that most CSS collection products are frequently updated and incrementally versioned, but perhaps Jesse may want to check into this for future CSS SIPs in the 'new style'? Thanks!

smclaughlin7 commented 1 year ago

@beatricemueller @c-suh Good news. The NSSDCA finished archiving this SIP to tape -- 256,910 products for 1.51 TB -- so we're done:

urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220513_delta_20230110154527571604::1.0

I just want to reiterate that we did not ingest these two collection products

urn:nasa:pds:gbo.ast.catalina.survey:data_partially_processed::176.0 urn:nasa:pds:gbo.ast.catalina.survey:data_raw::176.0

because the checksums listed in the SIP manifest file did not much the checksums for the downloaded files but we did ingest and archive all associated basic products listed in the SIP manifest file.

c-suh commented 9 months ago

@smclaughlin7 thank you for the update! @beatricemueller we are closing this ticket, but please resubmit those two collections that Stef mentions above.

beatricemueller commented 9 months ago

Catherine, I presume you mean

from Stef: I just want to reiterate that we did not ingest these two collection products

urn:nasa:pds:gbo.ast.catalina.survey:data_partially_processed::176.0 urn:nasa:pds:gbo.ast.catalina.survey:data_raw::176.0

because the checksums listed in the SIP manifest file did not much the checksums for the downloaded files but we did ingest and archive all associated basic products listed in the SIP manifest file.

These collections do not need to be re-submitted as they have been superseded by subsequent collections.

Beatrice

On Sep 29, 2023, at 13:22, Catherine Suh @.***> wrote:

@smclaughlin7 https://www.google.com/url?q=https://github.com/smclaughlin7&source=gmail-imap&ust=1696623735000000&usg=AOvVaw1wv8ZkNwRpjlvIzH6Trfee thank you for the update! @beatricemueller https://www.google.com/url?q=https://github.com/beatricemueller&source=gmail-imap&ust=1696623735000000&usg=AOvVaw0zSr5LegFjif7M_vgjMA40 we are closing this ticket, but please resubmit those two collections that Stef mentions above.

— Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/327%23issuecomment-1741447113&source=gmail-imap&ust=1696623735000000&usg=AOvVaw1FSw2FuQNo5AE6mgGQxOqB, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AKLIINEG726527HE6M43HSTX44UXNANCNFSM6AAAAAATTQ4N54&source=gmail-imap&ust=1696623735000000&usg=AOvVaw2nxFlWHZ99fOYnFZiDn-SY. You are receiving this because you were mentioned.

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

c-suh commented 9 months ago

Hi @beatricemueller - okay, will make note of this. Thank you for the reminder!