search

NASA-PDS / operations

Tickets for the PDSEN Operations Team
Other
5 stars 1 forks source link

[nssdca-delivery] PDS_ATM Pack11 (InSight final delivery) #446

Closed tinagueth closed 6 months ago

tinagueth commented 8 months ago

Discipline Node Information

NOTE: If you have multiple delivery packages, we strongly encourage you to submit these in batches of 3 to 10 per issue with one ZIP file of the packages and another ZIP file of the validation reports. Please use a descriptive title, such as "Node Mission misc batch #".

Engineering Node Process

See the internal EN process at https://pds-engineering.jpl.nasa.gov/content/nssdca_interface_process

oddes commented 8 months ago

@tinagueth : These 2 sets have been posted for NSSDCA processing. From tomorrow, you can check the status at https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp using the SIP LIDs below:

SIP LIDs: urn:nasa:pds:system_bundle:product_sip_deep_archive:insight_ps_v3.2_20231016 urn:nasa:pds:system_bundle:product_sip_deep_archive:insight_twins_v3.2_20231016

smclaughlin7 commented 8 months ago

@tinagueth : The NSSDCA's front-end processing failed these two SIP LIDs

because it could not validate the bundle products:

Remarks: Bundle failed schema validation: cvc-complex-type.2.4.a: Invalid content was found starting with element '{"http://pds.nasa.gov/pds4/pds/v1":doi}'. One of '{"http://pds.nasa.gov/pds4/pds/v1":keyword, "http://pds.nasa.gov/pds4/pds/v1":description}' is expected.

I double-checked the bundle products via PDS Validate and Oxygen Editor which issued this same error.

Both bundle products failed because they specify the \<doi> attribute in but use a version of the IM where that attribute was not yet implemented: PDS4_PDS_1800.xsd/.sch. \<doi> was first implemented in IM PDS4_PDS_1B00.xsd/.sch.

Possible solutions:

  1. Update both bundle products to use IM PDS4_PDS_1B00.xsd/.sch (or later), or
  2. Continue using IM PDS4_PDS_1800.xsd/.sch in both bundle products but delete the \<doi> from .

When resolved, please revalidate and resubmit both bundles to EN.

Let me know if you have any questions. Thanks!

jordanpadams commented 8 months ago

@tinagueth I see validate was using schemas housed at ATM versus the online schemas. Is that intentional? Were those schemas modified to include DOI information? All other validation runs I have tried using PDS4 v1800 throw errors, but your validation reports do now show these errors like they should.

tinagueth commented 8 months ago

@smclaughlin7 @jordanpadams this is the reply from Lyle "Jordan, I put in the tags “post-validation” so that the DOIs could be tracked even though it violates 1.8.0.0. I guess I can take the tags out. I’m not going to update the IM version. - Lyle"

What is the next step? Re-do deep-archive and validation reports as well? Thanks

jordanpadams commented 8 months ago

@tinagueth Let's revisit in a week? I have some offline discussions happening with NSSDCA to review whether or not this data can be accepted as-is.

tinagueth commented 8 months ago

@jordanpadams works for us (I will keep Lyle in the loop as he does all the prep work before I do the deep-archive part). Thanks.

smclaughlin7 commented 8 months ago

@smclaughlin7 @jordanpadams this is the reply from Lyle "Jordan, I put in the tags “post-validation” so that the DOIs could be tracked even though it violates 1.8.0.0. I guess I can take the tags out. I’m not going to update the IM version. - Lyle"

@tinagueth Would Lyle be open one of these solutions (both require revalidation and resubmission): 1) In the bundle products, keep the IM at 1.8.0.0, but temporarily remove the DOI attribute, revalidate the bundles, and generate new SIPs and submit to EN Github. Then after the NSSDCA archives the bundle, readd the DOI attribute. 2) For the bundle products only, permanently update the IM to 1.B.0.0, evalidate the bundles, and generate new SIPs and submit to EN Github.

jordanpadams commented 8 months ago

@smclaughlin7 I would like to pause on this submission until we discuss this further at our meeting tomorrow

temporarily remove the DOI attribute

This is not really how we should be managing an archive, so I don't necessarily want to encourage this unless it is absolutely necessary

smclaughlin7 commented 8 months ago

@jordanpadams Understood. I'm on vacation tomorrow (10/27 thru 11/6) so I will not be at our EN-NSSDCA meeting.

jordanpadams commented 7 months ago

@lylehuber @tinagueth per discussions with NSSDCA, the one file that needs to be valid from a schema perspective is the bundle labels, so can you please remove the DOI metadata from those labels?

lylehuber commented 7 months ago

This is actually the kind of thing we’re referring to when we suggest that when “2.0” arrives, we should plan to migrate everything to “2.0”. If you try to do something useful, it conflicts with someone’s version of what is the right thing to do.

OK, removing the DOI…

Lyle

jordanpadams commented 6 months ago

@smclaughlin7 The DOIs have been removed from this bundle so you should be good to go to reload this data. Should we resubmit through the website?

smclaughlin7 commented 6 months ago

@jordanpadams Our front-end already recorded the two SIP LIDVIDs for this InSight delivery #446 and will reject any SIP with the same LIDVIDs. #469 contains the replacement SIPs, with new LIDVIDs, for the two corrected InSight bundles.