search

NASA-PDS / operations

Tickets for the PDSEN Operations Team
Other
5 stars 1 forks source link

[nssdca-delivery] urn:nasa:pds:gbo.ast.catalina.survey::1.0 20220408 RESUBMITTED (see comment) #291

Closed beatricemueller closed 1 year ago

beatricemueller commented 2 years ago

Discipline Node Information

NOTE: If you have multiple delivery packages, we strongly encourage you to submit these in batches of 3 to 10 per issue with one ZIP file of the packages and another ZIP file of the validation reports. Please use a descriptive title, such as "Node Mission misc batch #".

Engineering Node Process

See the internal EN process at https://pds-engineering.jpl.nasa.gov/content/nssdca_interface_process

beatricemueller commented 2 years ago

@c-suh Note that the validation report might have errors, but any file that throws an error is pulled and is not submitted and will be submitted subsequently once fixed. This is an acceptable approach as discussed in #252

smclaughlin7 commented 2 years ago

@beatricemueller @jstone-psi @c-suh Hi Bea, MIght this submission have the same issue as #273 (SIP LIDVID urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220311_delta_20220623194937027150) where 7 products failed our ingest process because the files were moved from their original directory given in the SIP to subfolders named “SUPERSEDED” before the NSSDCA could ingest/download them? (I posted a list of the failed products in the comments for #273).

Our ingest process also failed some prodcuts in these two submissions for the same problem:

274 for SIP LIDVID urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220318_delta_20220624185620740018::1.0 -- 62 products failed

275 for SIP LIDVID urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220325_delta_20220627173110587170::1.0 -- 54 products failed

I'll post a list of these failed products as soon as the ingest team sends it.

Thanks.

smclaughlin7 commented 2 years ago

@c-suh Could you please wait until we hear from @beatricemueller before processing this submission? Thanks!

smclaughlin7 commented 2 years ago

@c-suh I'm revoking my hold request. Jesse confirmed this delivery should be good to go. Thanks! @jstone-psi @beatricemueller

c-suh commented 2 years ago

@beatricemueller and @jstone-psi, a friendly nudge to upgrade your Validate tool to the latest (currently v2.3.0). Thank you!

This set has been posted for NSSDCA processing. From tomorrow, you can check the status at https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp using the SIP LID below:

SIP LID:

smclaughlin7 commented 2 years ago

@beatricemueller @jstone-psi @c-suh The NSSDCA could not process this SIP LID:

because the SIP manifest label https://pds.nasa.gov/data/pds4/manifests/2022/gbo.ast.catalina.survey_v1.0_20220408_sip_delta_20220809202518009284_v1.0.xml gives the wrong URL for the SIP manifest table:

<manifest_url>https://pds.nasa.gov/data/pds4/manifests/2022/gbo.ast.catalina.survey_v1.0_20220408_sip_delta_20220809202518009284_None_v1.0.tab</manifest_url>

For some reason the URL contains a bogus string "None_" that should not be there. Do you know why this happened?

Since our front-end process has already registered the SIP LID, please generate and submit a new submission package with a new SIP LID. Thank you.

jstone-psi commented 2 years ago

Hi, Stef,

It looks like there was a bad null value check in the label (re)building code. I've updated the code, and I'll rebuild the submission package.

beatricemueller commented 2 years ago

Stef, once Jesse rebuilds the SIP, do you want me to open a new submission on github, or just let you and Katherine know in github that the SIP has been fixed?

Bea

On Aug 18, 2022, at 1:56 PM, Jesse Stone @.***> wrote:

Hi, Stef,

It looks like there was a bad null value check in the label (re)building code. I've updated the code, and I'll rebuild the SIP file soon.

— Reply to this email directly, view it on GitHub https://www.google.com/url?q=https://github.com/NASA-PDS/operations/issues/291%23issuecomment-1219954429&source=gmail-imap&ust=1661461004000000&usg=AOvVaw2r2NW9OP3PEVMH2tCTG6Nw, or unsubscribe https://www.google.com/url?q=https://github.com/notifications/unsubscribe-auth/AKLIINDCY3MGQ5GI6POOQTTVZ2PQXANCNFSM56MUTZQQ&source=gmail-imap&ust=1661461004000000&usg=AOvVaw0xVUgVOPQulsZuJiy4h6P5. You are receiving this because you were mentioned.

Beatrice E. A. Mueller Planetary Science Institute 1700 E. Ft. Lowell Rd., Suite 106 Tucson AZ 85719 mueller at psi . edu phone: 520-547-3950 FAX: 520-795-3697

smclaughlin7 commented 2 years ago

@jstone-psi Thank you for ingestigating why the was incorrect in the SIP label!

@c-suh Do you prefer @beatricemueller open a new submission on github or upload the new/revised submission package here #291? (I think it would be cleaner to open a new submission, but I defer to Catherine.)

c-suh commented 2 years ago

@beatricemueller and @smclaughlin7, since this is for a single package, I actually prefer uploading to this existing issue (either edit the main comment/description or add a link there to the comment with the revised package), but if this becomes problematic, I will work with what is given (just please indicate somewhere that the new submission is indeed resubmission of this one).

beatricemueller commented 2 years ago

@c-suh: Here is the corrected delivery package: https://sbnarchive.psi.edu/pds4/surveys/catalina_extras/sips/deltas/20220408.2_deltas.zip

c-suh commented 2 years ago

@beatricemueller, this resubmitted set has been posted for NSSDCA processing. From tomorrow, you can check the status at https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp using the SIP LID below:

SIP LID:

jstone-psi commented 1 year ago

Hi, Stef,

It looks like there was a bad null value check in the label (re)building code. I'll update it.

Thanks for notifying us!

--Jesse

On 8/18/22 9:54 AM, smclaughlin7 wrote:

@beatricemueller https://github.com/beatricemueller @jstone-psi https://github.com/jstone-psi @c-suh https://github.com/c-suh The NSSDCA could not process this SIP LID:

  • urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220408_delta_20220809202518009284

because the SIP manifest label https://pds.nasa.gov/data/pds4/manifests/2022/gbo.ast.catalina.survey_v1.0_20220408_sip_delta_20220809202518009284_v1.0.xml gives the wrong URL for the SIP manifest table:

|https://pds.nasa.gov/data/pds4/manifests/2022/gbo.ast.catalina.survey_v1.0_20220408_sip_delta_20220809202518009284_None_v1.0.tab|

For some reason the URL contains a bogus string |"None_"| that should not be there. Do you know why this happened?

Since our front-end process has already registered the SIP LID, please generate and submit a new submission package with a new SIP LID. Thank you.

— Reply to this email directly, view it on GitHub https://github.com/NASA-PDS/operations/issues/291#issuecomment-1219719086, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGF476LAQ5KHO7R5NNNNYUDVZZTFJANCNFSM56MUTZQQ. You are receiving this because you were mentioned.Message ID: @.***>

smclaughlin7 commented 1 year ago

@beatricemueller @jstone-psi @c-suh Hi Beatrice, Jesse: Good news! We successfully ingested all the products contained in this SIP LID (i.e., downloaded and registered all the products) and are archiving to tape now:

urn:nasa:pds:system_bundle:product_sip_deep_archive:gbo.ast.catalina.survey_v1.0_20220408_delta_20220819181155022968

I'd like to note that we detected several primary basic product LIDVIDs listed in one or more collection inventory files contained/delivered in this SIP but those products are not online and were not listed in this SIP (not a problem because they're not online). I attached a list of the LIDVIDs. If these products are not online, perhaps they should be removed from future collection inventories?

This does not affect our ingest process; we just remark on this finding. Thanks!

gbo.ast.catalina.survey_v1.0_20220408_delta_20220819181155022968_ProdsInCollsButNotOnline.txt

smclaughlin7 commented 1 year ago

Regarding comment about primary basic product LIDVIDs listed in collection inventories but the products are not online: Resolved; See Jesse's comment for #297:

Thanks, Stef! I think I looked into this issue previously, and the missing files had failed validation, so weren't being included moved to the archive, but they were being added to the collection anyway. I've since resolved this issue, so we should be able to filter them out of future collections.

c-suh commented 1 year ago

Hi @smlaughlin7! I was checking on the status of this package and see that it's been "Processed with Exception". I'm guessing that it's because of the "Invalid" status forurn:nasa:pds:gbo.ast.catalina.survey:document:: with the remark "Collection urn:nasa:pds:gbo.ast.catalina.survey:document not found in manifest." Is this something to take back to the node?

smclaughlin7 commented 1 year ago

Hi @smlaughlin7! I was checking on the status of this package and see that it's been "Processed with Exception". I'm guessing that it's because of the "Invalid" status for urn:nasa:pds:gbo.ast.catalina.survey:document:: with the remark "Collection urn:nasa:pds:gbo.ast.catalina.survey:document not found in manifest." Is this something to take back to the node?

@c-suh No, we don't have to take this back to PSI node.

PSI node runs a special version of the Deep Archive software that they've tweaked so that only the latest new version of each primary collection product and any new primary basic products are listed in the SIP manifest file, which we informally call a "delta SIP." (The standard Deep Archive software lists all primary collections and all their primary basic products contained in a bundle in the SIP manifest file. This method is too unwieldy for the Cat. Sky Survey bundle which has many tens of thousands of products appended every day. So PSI node kindly identifies only the products we need to archive for each submission.)

We/NSSDCA flagged primary collection product urn:nasa:pds:gbo.ast.catalina.survey:document:: as "Invalid" because it is identified in the bundle product but not included in the "delta SIP" manifest because nothing changed for the product, including primary basic products, since the last submission. This document collection does not change very often, so we can expect to see this Remark is future submissions.

c-suh commented 1 year ago

@beatricemueller this resubmitted package for gbo.ast.catalina.survey_v1.0_20220408_delta_20220819181155022968 has been archived!