search

NASA-PDS / operations

Tickets for the PDSEN Operations Team
Other
5 stars 1 forks source link

[nssdca-delivery] urn:nasa:pds:asurgcps_slides_jovian_system::1.0 and urn:nasa:pds:msl_apxs_supplement_sols_0_3076::1.1 #473

Closed gbowen99 closed 3 months ago

gbowen99 commented 6 months ago

Discipline Node Information

Updated by @jpadams: Archive.zip

NOTE: If you have multiple delivery packages, we strongly encourage you to submit these in batches of 3 to 10 per issue with one ZIP file of the packages and another ZIP file of the validation reports. Please use a descriptive title, such as "Node Mission misc batch #".

Engineering Node Process

See the internal EN process at https://pds-engineering.jpl.nasa.gov/content/nssdca_interface_process

c-suh commented 5 months ago

@gbowen99 and @smclaughlin7 this set has been posted for NSSDCA processing. From tomorrow, you can check the status at https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp using the SIP LID below:

SIP LID:

The other 2 sets, asurgcps_slides_jovian_system_v1.0_20231215 and msl_apxs_supplement_sols_0_3076_v1.1_20231030, have a number of validate errors. The errors for msl_apxs_supplement_sols_0_3076_v1.1 are identical to the errors for the set posted in https://github.com/NASA-PDS/operations/issues/468 and are found in asurgcps_slides_jovian_system_v1.0 as well, although this second set has additional errors to be addressed.

I will attach the validation reports below. Please address these errors then resubmit the updated delivery packages. Thank you!

As also mentioned in the other ticket, I've noticed that you're using an older version of Validate and highly recommend upgrading to the latest version as it has the latest features and bug-fixes. Thank you!

-Catherine

Validation reports:

smclaughlin7 commented 5 months ago

@gbowen99 @c-suh This SIP LID urn:nasa:pds:system_bundle:product_sip_deep_archive:msl_gt_diagenesis_supplement_v1.1_20231030 failed NSSDCA front-end processing because the SIP xml label specifies this bundle LIDVID msl_gt_diagenesis_supplement::1.1 whereas the SIP manifest file specifies this bundle LIDVID urn:nasa:pds:msl_gt_diagenesis_supplement::1.1. For some reason, the SIP xml label specifies the wrong/incomplete bundle LIDVID. Please correct this issue and generate a new Deep Archive Package for this bundle.

@c-suh @oddes Should @gbowen9 post the new Deep Archive Package to this ticket?

c-suh commented 5 months ago

Node's re-delivery of all packages, including the submitted msl_gt_diagenesis_supplement_v1.1, is currently blocked by this deep-archive bug.

jordanpadams commented 4 months ago

@c-suh see updated packages attached above.

c-suh commented 4 months ago

@gbowen99 the 3 packages which @jordanpadams updated while the deep archive bug is being addressed have been posted for NSSDCA processing! From tomorrow, you can check the status at https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp using the SIP LIDs below:

SIP LIDs: urn:nasa:pds:system_bundle:product_sip_deep_archive:asurgcps_slides_jovian_system_v1.0_20240209 urn:nasa:pds:system_bundle:product_sip_deep_archive:msl_apxs_supplement_sols_0_3076_v1.1_20240209 urn:nasa:pds:system_bundle:product_sip_deep_archive:msl_gt_diagenesis_supplement_v1.1_20240209

gbowen99 commented 4 months ago

@c-suh Understood, thank you for the update.

smclaughlin7 commented 4 months ago

@c-suh @jordanpadams Our NSSDCA front-end process failed this SIP LID

urn:nasa:pds:system_bundle:product_sip_deep_archive:msl_gt_diagenesis_supplement_v1.1_20240209

because the bundle product located at https://pds-geosciences.wustl.edu/msl/urn-nasa-pds-msl_gt_diagenesis_supplement/bundle_msl_gt_diagenesis_supplement.xml does not match the checksum listed in the SIP manifest https://pds.nasa.gov/data/pds4/manifest/2023/msl_gt_diagenesis_supplement_v1.1_20231030_sip_v1.0.tab. I inspected readme.txt and it's checksum differs from the one in the SIP manifest. I did not

Please generate and submit a new SIP Deep Archive package for the current online version of this bundle. The front-end did not detect any issues in the other two SIPs so it sent them to Ingest.

Thanks!

jordanpadams commented 4 months ago

@c-suh see attached. msl_gt.zip

c-suh commented 4 months ago

@jordanpadams and @smclaughlin7 the new package for the following SIP has been posted!

urn:nasa:pds:system_bundle:product_sip_deep_archive:msl_gt_diagenesis_supplement_v1.1_20240209

c-suh commented 4 months ago

Statuses Submitted to Ingest:

Failed and reposted for NSSDCA:

smclaughlin7 commented 4 months ago

@c-suh Our front-end process not pull this "reposted" SIP

because the LIDVID did not change when the manifest file was corrected (to correct the checksum listed for the bundle product). The front-end had already failed that SIP LIDVID on Feb 11 when it detected the bundle checksum mismatch. Please generate and post a new SIP with a new LIDVID. That'll trigger our process to pull it. Thanks!

c-suh commented 4 months ago

@jordanpadams :point_up:. Not sure how imminent the deep archive release is and so if this is for you or for @gbowen99 to address?

jordanpadams commented 4 months ago

@c-suh here is the new package. I just updated the SIP/AIP labels version_id to 2.0. msl_gt_v2.0.zip

c-suh commented 4 months ago

@jordanpadams please confirm that these files are correct as I'm not sure where the 2.0 is supposed to be reflected and there seems to be a discrepancy or two.

Observations from looking at the posted msl_gt_diagenesis_supplement files (timestamp of 10:59am) in the 2024 directory:

jordanpadams commented 4 months ago

@c-suh

filenames are msl_gt_diagenesis_supplement_v1.120240209*_v1.0.{tab|xml}

This doesn't really matter, but updated here: msl_gt_v2.0.zip

in the aip XML

  • Identification_Area.version_id is 2.0 Identification_Area.Modification_History.Modification_Detail.version_id is 1.0 in the sip XML, version_ids are opposite of what's written in the aip XML Identification_Area.version_id is 1.0 Identification_Area.Modification_History.Modification_Detail.version_id is 2.0

Ideally I would have updated the Modification Detail, but this doesn't actually matter in terms of the validity of the files, so I am just leaving them. I definitely don't have the time/patience to be an archivist and do this manual editing of files :-)

c-suh commented 4 months ago

@jordanpadams if the package from this morning is fine, meaning the filenames and version id inconsistencies don't matter (the Identification_Area.version_id and Identification_Area.Modification_History.Modification_Detail.version_id pair in the aip being different from that in the sip), then I will leave those files for NSSDCA to pick up. The package you attached this afternoon failed validation because the URIs (and filenames?) for the manifest TABs were not updated in the aip and sip XMLs. I could fix those, but it does seem better to keep the manual editing of files to a minimum :]

jordanpadams commented 4 months ago

@c-suh i will look at updating these files on Monday.

jordanpadams commented 4 months ago

@c-suh see attached msl_gt_v2.0.zip

jordanpadams commented 4 months ago

@gbowen99 You should be able to upgrade your deep-archive software and continue your deep archive deliveries. This specific delivery should be good to go. Let us know if you run into additional issues.

c-suh commented 4 months ago

@jordanpadams and @smclaughlin7 the latest package for v2.0 of the following SIP LID has been posted. Note that a v1.0 was posted last Friday the 23rd; please let me know if I should delete those files.

SIP LID:

gbowen99 commented 4 months ago

@jordanpadams thanks for the update!

smclaughlin7 commented 4 months ago

@jordanpadams and @smclaughlin7 the latest package for v2.0 of the following SIP LID has been posted. Note that a v1.0 was posted last Friday the 23rd; please let me know if I should delete those files. SIP LID:

  • urn:nasa:pds:system_bundle:product_sip_deep_archive:msl_gt_diagenesis_supplement_v1.1_20240209

@c-suh @jordanpadams Our front-end process did not pull this v2.0 package because the SIP LIDVID specified in the SIP label is one we've pulled before:

Our nightly SIP Puller checks to see if we've already got the LIDVID before downloading the SIP. In this case SIP https://pds.nasa.gov/data/pds4/manifest/2024/msl_gt_diagenesis_supplement_v1.1_20240209_sip_v2.0.xml did not get pulled over at all. (A remark in our internal log for SIP Puller indicates this.)

Recall our front-end process received and failed this SIP LIDVID on 2024-02-11 because the bundle product located at https://pds-geosciences.wustl.edu/msl/urn-nasa-pds-msl_gt_diagenesis_supplement/bundle_msl_gt_diagenesis_supplement.xml did not match checksum in manifest. See my post above from ~2 weeks ago.

Please change the SIP LIDVID so that it's unique, then make and post a new Deep Archive package. Thanks!

p.s. - It's OK to keep the V1.0 files posted last Friday the 23rd. The SIP LIDVID's the same there, too, so our SIP Puller ignores 'em.

c-suh commented 3 months ago

@jordanpadams how would you like to proceed for this v2.0?

jordanpadams commented 3 months ago

@gbowen99 per @smclaughlin7's comment above, the checksums for the MSL GT Diagenesis bundle and readme do not match what is loaded into the Registry. It appears someone modified those files after they were loaded into the Registry. Please reload that bundle into the registry with the correct checksums, re-run pds-deep-registry-archive, and submit a new ticket for that delivery. (Note: Be sure to upgrade PDS Deep Archive prior to running again.

File Registry Checksum Actual Checksum
bundle_msl_gt_diagenesis_supplement.xml 20dfa377f9eed3ade2084baed8226d21 9c0c5d6489e871567fe749172c4e25d9
readme.txt 933bb5b10e1f14b4e36e0df4c5431c83 5d25a42641b66732f7f4c88dfdb0cb28
jordanpadams commented 3 months ago

Note: I removed urn:nasa:pds:msl_gt_diagenesis_supplement::1.1 from this delivery to avoid confusion with the future re-delivery

c-suh commented 3 months ago

The resubmission for msl_gt_diagenesis_supplement_v1.1 is in https://github.com/NASA-PDS/operations/issues/499.

@gbowen99 the other 2 packages listed below have been validated by NSSDCA, submitted to Ingest, and Ingested. NSSDCA has now taken responsibility.