Add rules and tests for `opus_id` & `opus_id_to_primary_logical_path` for uranus occ earthbased bundle set

[mace-space]

Comments updated by Dave on 04/05/24

• Fixes #12
• Fixes #13

Reference: Rob's email OPUS Import and PdsFile rules for Uranus occultations from 08/22/2023, 1:15PM

Description of changes:

• Add opus_id rules for uranus_occs_earthbased
  • Store bundle prefix and opus id prefix to prefix_mapping, and use it to create rules for different opus ids
• Add opus_products rules for uranus_occs_earthbased with these changes different from reference email:
  • atmosphere occ rules will recognize the rms index file with either *_atmos_index_* or *_atmosphere_index_* in the file name.
  • ring-plane* products will be removed from rings & atmosphere occs
  • Remove "Global Ring Orbital Fit" for atmosphere occ
  • Remove "Occultation Ring Model" for global occ
• Add opus_type rules for uranus_occs_earthbased
• Add opus_id_to_primary_logical_path rules for uranus_occs_earthbased
• Add tests for opus_id, opus_type, opus_products, and opus_id_to_primary_logical_path
  • Add test_opus_products and test_opus_id_to_primary_logical_path in rules/uranus_occs_earthbased.py
  • Add test_opus_id and test_opus_type in tests/test_pds4file_blackbox.py
• Make sure there is a 1-to-1 mapping between opus id and primary filespec
  • Store all primary file spec in uranus_occs_earthbased_primary_filespec.py and run this test for each file:
  • Create a pdsfile instance using from_logical_path (path of a primary filespec)
  • Call opus_id to get the corresponding opus id from the instance of step 1
  • Use that opus id from step 2 to create another pdsfile instance by calling from_opus_id
  • Compare the logical path of both pdsfile instances from step 1 & 3, and they will be the same (the path of primary filespec)
• Replace all holdings to cls.PDS_HOLDINGS in pdsfile.py

Know problems:

• Missing rules & tests for preview and diagram images
• Review of the descriptions in opus_type rules

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 78.68%. Comparing base (01bf2d9) to head (1143f1e). Report is 1 commits behind head on main.

:exclamation: Current head 1143f1e differs from pull request most recent head 8d2fe73. Consider uploading reports for the commit 8d2fe73 to get more accurate results

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #25 +/- ## ========================================== + Coverage 78.35% 78.68% +0.32% ========================================== Files 39 40 +1 Lines 5720 5719 -1 Branches 1247 1374 +127 ========================================== + Hits 4482 4500 +18 + Misses 1015 1001 -14 + Partials 223 218 -5 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[rfrenchseti]

It's going to be an immense pain to make the opus_id Translator as complete as you've done for KAO, and I don't think it's necessary. The only reason we need that Translator is to convert between a "primary filespec" and an opus_id. But the "primary filespec" is only taken from the index file. Thanks to Emilie's index file generator, we can control what files are listed in the index file, and we've decided that only a single resolution (the best) of each ring plane and the atmosphere will be included. This means we don't care about browse images, or counts-v-time in the rings. All we care about is:

• u0_kao_91cm_734nm_radius_<ring>_<ie>_100m.xml
• global/u0_kao_91cm_734nm_radius_equator_<ie>_100m.xml
• atmosphere/u0_kao_91cm_734nm_counts-v-time_atmos_<ie>.xml

[rfrenchseti]

Actually ignore a lot of what I said above...we'll discuss in the meeting today.

[juzen2003]

Status:

• Add opus id rules for uranus occ bundles (Done)
• Add opus id to primary file spec rules (Work in progress)
• Add tests for above rules (work in progress)

Here are some bundles that have missing opus id abbrev mapping in format doc or different detector values:

• u137_hst_fos

  • No opus id abbrev mapping in the format table
  • Detector: Wide Field Camera 3
  • Current opus id prefix placeholder: hstfos-wfc3-occ-1996-076-u137

• u138_hst_fos

  • No opus id abbrev mapping in the format table
  • Detector: Wide Field Camera 3
  • Current opus id prefix placeholder: hstfos-wfc3-occ-1996-101-u138

• u144_caha_123cm

  • The context.csv has this urn:nasa:pds:context:telescope:caha.1m23 but the format table has caha-calar_alto.1m23 instead of just caha.1m23, not sure if we want the value to match just like other bundles.
  • Detector: 'Generic NICMOS IR Camera', so I put ir as well in opus id, let me know if we want a different value like nicmos-ir for this.

• Note: These two bundles has different detectors other than ir and vis

  • u14_opmt_106cm: gaas
  • u149_lowell_180cm: ccd

[juzen2003]

For record, here is the start date (yod) for each bundle

{'u0201_palomar_508cm': ['2002-210'],
 'u0_kao_91cm': ['1977-069'],
 'u102a_irtf_320cm': ['1992-190'],
 'u102b_irtf_320cm': ['1992-190'],
 'u103_eso_220cm': ['1992-193'],
 'u103_palomar_508cm': ['1992-193'],
 'u1052_irtf_320cm': ['1988-133'],
 'u11_ctio_400cm': ['1980-080'],
 'u12_ctio_400cm': ['1980-229', '1980-228'],
 'u12_eso_104cm': ['1980-229', '1980-228'],
 'u12_eso_360cm': ['1980-229', '1980-228'],
 'u12_lco_250cm': ['1980-229', '1980-228'],
 'u134_saao_188cm': ['1995-252'],
 'u137_hst_fos_540nm': ['1996-076'],
 'u137_irtf_320cm': ['1996-076'],
 'u138_hst_fos_540nm': ['1996-101'],
 'u138_palomar_508cm': ['1996-101'],
 'u13_sso_390cm': ['1981-116'],
 'u144_caha_123cm': ['1997-273'],
 'u144_saao_188cm': ['1997-273'],
 'u149_irtf_320cm': ['1998-310'],
 'u149_lowell_180cm': ['1998-310'],
 'u14_ctio_150cm': ['1982-112'],
 'u14_ctio_400cm': ['1982-112'],
 'u14_eso_104cm': ['1982-112'],
 'u14_lco_100cm': ['1982-112'],
 'u14_lco_250cm': ['1982-112'],
 'u14_opmt_106cm': ['1982-112'],
 'u14_opmt_200cm': ['1982-112'],
 'u14_teide_155cm': ['1982-112'],
 'u15_mso_190cm': ['1982-121'],
 'u16_palomar_508cm': ['1982-155'],
 'u17b_saao_188cm': ['1983-084'],
 'u23_ctio_400cm': ['1985-124'],
 'u23_mcdonald_270cm': ['1985-124'],
 'u23_teide_155cm': ['1985-124'],
 'u25_ctio_400cm': ['1985-144'],
 'u25_mcdonald_270cm': ['1985-144'],
 'u25_palomar_508cm': ['1985-144'],
 'u28_irtf_320cm': ['1986-116'],
 'u2_teide_155cm': ['1977-357'],
 'u34_irtf_320cm': ['1987-057'],
 'u36_ctio_400cm': ['1987-092'],
 'u36_irtf_320cm': ['1987-092', '1987-089'],
 'u36_maunakea_380cm': ['1987-092', '1987-089'],
 'u36_sso_230cm': ['1987-092'],
 'u36_sso_390cm': ['1987-092'],
 'u5_lco_250cm': ['1978-100'],
 'u65_irtf_320cm': ['1990-172'],
 'u83_irtf_320cm': ['1991-176'],
 'u84_irtf_320cm': ['1991-179'],
 'u9539_ctio_400cm': ['1993-181'],
 'u9_lco_250cm': ['1979-161']}

[rfrenchseti]

• HST FOS (Faint Object Spectrograph) has nothing to do with WFC3. It's a totally separate instrument. They didn't even exist at the same time. FOS was an original instrument, replaced by NICMOS during the 1997 servicing missing. WFC3 is a 4th generation instrument that was installed during the 2009 servicing mission. The OPUS ID prefix should be hst-fos. As part of adding this bundle to OPUS, I've added an HST FOS instrument to go along with HST ACS, etc. I'll add this to the OPUS ID documentation.
• The problem with caha.1m23 sounds like a problem with the bundle. That's not a valid telescope name. See the valid names in https://pds.nasa.gov/data/pds4/context-pds4/telescope/ You should talk to Mia and see if this came up when she was validating the bundle. We can certainly add this alias to the table (and to OPUS) to handle the bad bundle.
• For "Generic NICMOS IR" isn't the OPUS ID prefix just hst-nicmos? You shouldn't need an ir. NICMOS is the detector.
• For the last two bundles, you can just put gaas and `ccd.
• I'm a little confused about the statement "other than ir or vis". Aren't we using insb for most of the generic IR detectors?

[mace-space]

• u144_caha_123cm

Yes, as Rob said there’s an error in the LID reference, it should be urn:nasa:pds:context:telescope:caha-calar_alto.1m23 (and urn:nasa:pds:context:facility:observatory.caha-calar_alto).

Note: there are both facility and telescope context products listed in collection_context.csv. In the case of caha 123, there was an error in the telescope LID (the facility LID was valid), but in the case of Pic du Midi there was an error in the facility LID (should be urn:nasa:pds:context:facility:observatory.pic_du_midi rather than urn:nasa:pds:context:facility:observatory.pic_du_midi-bigorre) and the telescope LID was correct. Since I think you've been referring to the telescope context products, the Pic du Midi facility error didn’t appear.

I've been working on fixing these LID references (and some other issues) and updating the bundles so there will be another version coming up.

[juzen2003]

Thanks for the comment, I'll update them.

• I'm a little confused about the statement "other than ir or vis". Aren't we using insb for most of the generic IR detectors?

Sorry for the confusing comment, I meant to say vis, ir, and insb. I forgot insb, it's the most common one.

• HST FOS (Faint Object Spectrograph) has nothing to do with WFC3. It's a totally separate instrument. They didn't even exist at the same time. FOS was an original instrument, replaced by NICMOS during the 1997 servicing missing. WFC3 is a 4th generation instrument that was installed during the 2009 servicing mission. The OPUS ID prefix should be hst-fos. As part of adding this bundle to OPUS, I've added an HST FOS instrument to go along with HST ACS, etc. I'll add this to the OPUS ID documentation.

For these two hst-fos bundles, I was confused by the tag in the data label. They are something like this: (so I thought it was wfc3 abbrev)

<Observing_System_Component>
<name>Wide Field Camera 3</name>
<type>Instrument</type>

Also, just would like to point it out that they don't have telescope in collection_context.csv, not sure if this is expected.

[rfrenchseti]

Well I think you found another bug in the bundles. This is clearly wrong (especially since WFC3 didn't even exist in 1996):

      <Observing_System_Component>
        <name>Wide Field Camera 3</name>
        <type>Instrument</type>
        <Internal_Reference>
          <lid_reference>urn:nasa:pds:context:instrument:hst.fos</lid_reference>
          <reference_type>is_instrument</reference_type>
        </Internal_Reference>
      </Observing_System_Component>

Hopefully @mace-space can confirm and fix. For now, just assume the LID is correct and the text is wrong.

I don't think you need telescope in the collection_context.csv file because this observation didn't use a telescope...those are ground-based. It used HST, and they do have:

S,urn:nasa:pds:context:instrument:hst.fos
S,urn:nasa:pds:context:instrument_host:spacecraft.hst

[mace-space]

Good spot on the bug for HST FOS – yes, those data labels should not say Wide Field Camera 3. I've added that to the list of things to fix, thanks.

[juzen2003]

Updates:

• Add rules for opus_id_to_primary_logical_path
• Add tests for opus_id_to_primary_logical_path (make sure a pdsfile instance from from_opus_id will return its corresponding primary filespec path when calling logical_path).
• Put all primary filespec to dropbox Pds4FileTest/pds4-holdings/bundles

[juzen2003]

Update:

• Update primary filespec list to have .xml files, and update the opus_id_to_primary_logical_path rule as well
• Update os_path_exists and os_path_isdir to check filesystem before return False when SHELVES_ONLY is set to True. This will make sure a pdsfile is cached as long as it exists in the filesystem. This is to fix from_path when calling on an existing pdsfile with SHELVES_ONLY set to True & missing shelves file.

[juzen2003]

Update:

• Revert the change to check filesystem when shelves files are missing with SHELVES_ONLY set to True
• Add rules & tests for opus_type
• Add rules $ tests for opus_products

Pending Items:

• Review the descriptions in opus type rules
• Will have to update rules & tests when preview and diagram images are available

[juzen2003]

Update opus_products rules:

• atmosphere occ rules will recognize the rms index file with either *_atmos_index_* or *_atmosphere_index_* in the file name.
• ring-plane* products will be removed from rings & atmosphere occs
• "Occultation Ring Model" products will be included for global occ

[juzen2003]

Update opus_products rules to

• Remove "Global Ring Orbital Fit" for atmosphere occ
• Remove "Occultation Ring Model" for global occ