search

NASA-PDS / operations

Tickets for the PDSEN Operations Team
Other
5 stars 1 forks source link

[nssdca-delivery] urn:nasa:pds:insight_documents::2.0 and urn:nasa:pds:insight_hp3_tem::1.0 and urn:nasa:pds:insight_rad::2.1 and urn:nasa:pds:insight_rise_derived::1.0 and urn:nasa:pds:insight_rise_raw::2.0 and urn:nasa:pds:insight_seis::3.0 #511

Open gbowen99 opened 6 months ago

gbowen99 commented 6 months ago

Discipline Node Information

gbowen99 commented 6 months ago

The Geosceinces node is aware of the errors present in the validation reports for urn:nasa:pds:insight_documents::2.0 and urn:nasa:pds:insight:seis::3.0. We are not planning on correcting them at this time.

smclaughlin7 commented 6 months ago

@c-suh @jordanpadams Could you please hold off checking and posting this SIP LID?

I see extraneous data files in the SIP manifest. I highlighted those files in the attached XLS version of the manifest: insight_documents_v2.0_20240429_sip_v1.0.tab.xlsx. The extraneous files are for collections having secondary memebership in the bundle and basic products in the secondary collections.

The only files that should be listed in a PDS4 SIP manifest are:

  1. files of the bundle product (e.g., bundle.xml, readme.txt)
  2. files of collection products (e.g., collection.xml and its inventory) having primary membership in the bundle
  3. files of basic products having primary membership in those collections

I have not see this for SIP manifests for ATM's Mars2020 MEDA bundle (https://pds-atmospheres.nmsu.edu/PDS/data/PDS4/Mars2020/mars2020_meda/bundle.xml). That bundle identifies several collections with secondary membership. To date, ATM has submitted it eight times and the manifests never included secondary collection products and their basic products.

Could this have something to do with how a node generates the Deep Archive package, e.g., via local file system or via registry? Should I open a new ticket for Deep Archive for this case?

Thanks.

jordanpadams commented 6 months ago

@smclaughlin7 yes. this is because they list them as secondary but have the files on the file system. we will have to file a ticket for this.

@nutjob4life can you create a new ticket for this and investigate? I would work with @jshughes to understand more details about primary vs. secondary products/collections.

jordanpadams commented 6 months ago

@nutjob4life this may also be something we didn't think of with pds-deep-registry-archive and accessing that information from the API. I don't think the API cares about primary vs. secondary, so it may cause some issues...

nutjob4life commented 6 months ago

Hi @jordanpadams, my access to my laptop is restored (thanks JPL help desk—took about an hour) and I will set up the ticket.

Reminder: I'm on vacation today through May 24th.

c-suh commented 6 months ago

@gbowen99 thank you for the note about the errors for documents 2.0 and seis 3.0. As @smclaughlin7 has requested, I have not posted the package for documents 2.0 but the other 4 sets have been posted for NSSDCA processing. From tomorrow, you can check the status at https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp using the SIP LIDs below:

SIP LIDs:

For the last package for rise, there are a number of validate errors (see attached report). Please let me know if these are errors you will be addressing or ignoring, such as done with documents 2.0 and seis 3.0. Thank you.

gbowen99 commented 5 months ago

@c-suh sorry for the late response!

I am a bit confused by the report you attached for the rise_raw dataset. Is that a validation report that you put together? The report I submitted for the bundle used validate 3.5.0-SNAPSHOT and doesn't have any errors present. Any idea what might be going on?

c-suh commented 5 months ago

Hi @gbowen99 - my apologies for the delay. Yes, the report that I attached above is one that I created. We ask the nodes to run the Validate tool with certain parameters, and then we at the Engineering node run the Validate tool again with different parameters. Since a new version of Validate was released last week, I did this again and see the same errors. I will check with our developers to make sure this isn't a bug on our part and will get back to you. Thank you for your patience!

c-suh commented 5 months ago

@gbowen99 this is an issue with our previous deep archive tool. If you have not already, please install the latest version, which has the bug fix for this error, to use that moving forward. For the problematic insight_rise_raw package, we will address the error then post this for NSSDCA processing; there is nothing more for you to do regarding this ticket for the time being (I expect documents 2.0 will go on another ticket after we implement the necessary functionality to the deep archive tool). Thank you!

smclaughlin7 commented 4 months ago

@c-suh @jordanpadams @gbowen99 Just an FYI --- The NSSDCA successfully ingested this SIP

But I'd like to note that the SIP manifest file contained 6 files for 3 extraneous, secondary product LIDs, due to a bug in the Deep Archive Tool when using the Registry to make the SIP:

Fortunately, our ingest process failed these products because their LIDs did not conform with the bundle product's LID, urn:nasa:pds:insight_rad.

We suggessfully ingested all the other products in the SIP, so @gbowen99 does not need to generate and submit a new SIP for the InSight Rad Bundle v2.1. Yay!

jordanpadams commented 4 months ago

@smclaughlin7 @gbowen99 we will reprocess and resubmit these as soon as we have a fix to deep archive

smclaughlin7 commented 4 months ago

@jordanpadams On second thought, reprocessing and resubmitting these as soon as you have a fix to deep archive makes sense. Would help confirm deep archive is working a intended when it uses the Registry to build SIPs. Thanks!!

jordanpadams commented 4 months ago

Blocked by https://github.com/NASA-PDS/deep-archive/issues/164, which is blocked by https://github.com/NASA-PDS/registry/issues/185

jordanpadams commented 2 months ago

@c-suh newly generated submission for insight_rad bundle: insight_rad_package.zip

smclaughlin7 commented 2 months ago

@jordanpadams Confirming that although the NSSDCA successfully ingested this SIP (for this case, ingest ignored the 3 extraneous, secondary product LIDs) urn:nasa:pds:system_bundle:product_sip_deep_archive:insight_rad_v2.1_20240429 you still want @c-suh to cross-check and post this revised SIP, without the extraneous secondary products, for NSSDCA to ingest and archive, correct? (Makes sense; this should confirm everything's working as expected now.)

jordanpadams commented 2 months ago

@smclaughlin7 in that case, if this is good enough, we will move forward with other submissions and follow-up on this issue if it appears again. thanks for the heads up!

smclaughlin7 commented 2 months ago

@jordanpadams I recommend re-opening this ticket because the NSSDCA have not yet finished ingesting this SIP LID:

Thanks!

smclaughlin7 commented 2 months ago

@jordanpadams Thanks for reopening this ticket. Looking at the history above, we decided not to submit these two SIPs delivered via this ticket because we knew the manifest files contained extraneous secondary products:

urn:nasa:pds:system_bundle:product_sip_deep_archive:insight_documents_v2.0_20240429 urn:nasa:pds:system_bundle:product_sip_deep_archive:insight_rise_raw_v2.0_20240429

Is it OK for GEO (@alknight-wustl or @gbowen99) to generate and submit new Deep Archive packages for the InSight Documents and raw RISE bundles, or do you/EN want to do that?

Thanks!

jordanpadams commented 2 months ago

@c-suh see the latest updated packages for all these SIPs for insight_rad, insight_documents, and insight_rise_raw here: insight_package.zip

@smclaughlin7 I update the package to include both.

c-suh commented 2 months ago

@gbowen99 and @smclaughlin7 the 3 sets provided by Jordan have been posted for NSSDCA processing! From tomorrow, you can check the status at https://nssdc.gsfc.nasa.gov/psi/ReportPDS4.jsp using the SIP LIDs below:

SIP LIDs: urn:nasa:pds:system_bundle:product_sip_deep_archive:insight_documents_v2.0_20240903 urn:nasa:pds:system_bundle:product_sip_deep_archive:insight_rad_v2.1_20240903 urn:nasa:pds:system_bundle:product_sip_deep_archive:insight_rise_raw_v2.0_20240903