search

NASA-PDS / operations

Tickets for the PDSEN Operations Team
Other
5 stars 1 forks source link

[nssdca-delivery] urn:nasa:pds:voyager1_rss_jupiter_raw::1.0 and urn:nasa:pds:voyager2_rss_jupiter_raw::1.0 #490

Open mace-space opened 4 months ago

mace-space commented 4 months ago

Discipline Node Information

NOTE: voyager2_rss_jupiter_raw_v1.0_20240228_sip_v1.0.tab contained invalid URLS ( https://pds-rings.seti.org/pds4/bundles/voyage22_rss_jupiter_raw//), which I corrected to https://pds-rings.seti.org/pds4/bundles/voyager2_rss_jupiter_raw//

Engineering Node Process

See the internal EN process at https://pds-engineering.jpl.nasa.gov/content/nssdca_interface_process

c-suh commented 3 months ago

@mace-space, these 2 sets have been posted for NSSDCA processing! From tomorrow, you can check the status at https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp using the SIP LIDs below:

SIP LIDs: urn:nasa:pds:system_bundle:product_sip_deep_archive:voyager1_rss_jupiter_raw_v1.0_20240228 urn:nasa:pds:system_bundle:product_sip_deep_archive:voyager2_rss_jupiter_raw_v1.0_20240228

mace-space commented 3 months ago

Thanks very much @c-suh!

Voyager 1 reports "Ready for Ingest", but Voyager 2 is reporting as failed with remark: "Manifest checksum calculated does not match manifest checksum in SIP". Could this be due to my correcting the URLS in voyager2_rss_jupiter_raw_v1.0_20240228_sip_v1.0.tab (as noted above) or is there some other issue?

jordanpadams commented 3 months ago

@mace-space correct. unfortunately, manually edit of just about anything with these products is very difficult because of all the moving parts. I would just re-load the data into the registry to represent what you want, and then re-run the deep archive software. sorry for the inconvenience. We will discuss with NSSDCA possibly removing some of these checksums from our files to avoid these collisions.

mace-space commented 3 months ago

Thanks. Here's the updated Vgr2: voyager2_rss_jupiter_raw_v1.0_NSSDCA_20240313.tar.gz

mace-space commented 3 months ago

Also, I checked Vgr1 (https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp with search term "urn:nasa:pds:system_bundle:product_sip_deep_archive:voyager1_rss_jupiter_raw_v1.0_20240228") and it's reporting “Experiment or Instrument not found” for urn:nasa:pds:context:instrument:vg1.eng and urn:nasa:pds:context:instrument:dsn.rss? Although those context products do exist: https://pds.nasa.gov/data/pds4/context-pds4/instrument/vg1.eng_1.0.xml, https://pds.nasa.gov/data/pds4/context-pds4/instrument/dsn.rss_1.0.xml

jordanpadams commented 3 months ago

@mace-space thanks! @c-suh see above.

@smclaughlin7 see above for possible processing issue with LIDs?

smclaughlin7 commented 3 months ago

@jordanpadams @mace-space

and it's reporting “Experiment or Instrument not found” for urn:nasa:pds:context:instrument:vg1.eng and urn:nasa:pds:context:instrument:dsn.rss? Although those context products do exist...

This type of remark, exper/instr not found, is for NSSDCA internal purposes and does not affect ingest of this delivery. (Yay!)

Details: We key PDS4 instrument host and instrument LIDs to the corresponding spacecraft and experiment records in our NSSDCA Master Catalog database so that our front-end process can automatically associate our spacecraft and experiments to the resulting NSSDCA Collection IDs by using context LIDs it finds in PDS4 bundle and collection products. Several times a year we slurp PDS4 context products, looking for new LIDs to insert in our database. Looks like it's time we do another one of these exercises! Once we do that, we'll manually assign the appropriate spacecraft and experiments to the NSSDCA Collection IDs associated with this VG1 delivery, e.g., https://nssdc.gsfc.nasa.gov/nmc/dataset/display.action?id=PSPA-01110, if not already assigned.

c-suh commented 3 months ago

@mace-space this corrected package has been posted for NSSDCA processing! From tomorrow, you can check the status at https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp using the SIP LID below:

SIP LID:

The package for voyager1 has been validated by NSSDCA, submitted to Ingest, and Ingested. NSSDCA has now taken responsibility for that package.

c-suh commented 2 months ago

The package for voyager2 has been submitted to ingest and is in pre-ingest.

smclaughlin7 commented 1 month ago

@mace-space We ingested this SIP LID urn:nasa:pds:system_bundle:product_sip_deep_archive:voyager2_rss_jupiter_raw_v1.0_20240313::1.0. However we only ingested the bundle products and its primary collection products because they were the only items listed in the SIP manifest file, https://pds.nasa.gov/data/pds4/manifests/2024/voyager2_rss_jupiter_raw_v1.0_20240313_sip_v1.0.tab. None of the primary basic products in that bundle were included in the manifest. Do you want to generate and submit a new Deep Archive/SIP package for the entire bundle?

The other SIP LID in the ticket urn:nasa:pds:system_bundle:product_sip_deep_archive:voyager1_rss_jupiter_raw_v1.0_20240228 has same issue, too.

We've also seen the same issue with several recent SIPs from GEO, so we've alerted @jordanpadams @c-suh. We have not seen this issue with recent SIPs from ATM.

jordanpadams commented 1 month ago

@mace-space we will take a look at this and resubmit for you

jordanpadams commented 1 month ago

@c-suh here is a new package to post: voyager2_rss_jupiter_raw_v1.0_20240509.tar.gz

@mace-space When you ran deep-archive, was it "soon" after the data was registered? If so, unfortunately, in order for this utility to work as expected with the Registry, we have some background utilities that run at a ~6 hour cadence to update some metadata so we can track bundles->collections->products effectively. Unfortunately, there is no real way to link bundles->products directly without running some other software to figure this out.

To be safe, we recommend waiting 8 hours after registering the data before running the pds-deep-registry-archive software. Sorry for the inconvenience. We will update the documentation to reflect this, and this will hopefully all go away when we are able to run the software for you.

mace-space commented 1 month ago

Thanks, @jordanpadams and @c-suh.

I'll make sure to wait before running pds-deep-registry-archive in future.

smclaughlin7 commented 1 month ago

@jordanpadams Thanks for making a new package for the Voyager2 RSS Jupiter Raw V1.0 bundle. The SIP manifest file now includes basic products. 👍

@mace-space @jordanpadams We recently ingested the other SIP LID in this ticket _urn:nasa:pds:system_bundle:product_sip_deep_archive:voyager1_rss_jupiter_raw_v1.020240228 but it only contained the bundle and collection products. Please make a new package for the Voyager2 RSS Jupiter Raw V1.0 bundle to ensure we deep-archive the entire bundle. Thanks!