Closed mace-space closed 4 months ago
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 78.68%. Comparing base (
01bf2d9
) to head (1143f1e
). Report is 1 commits behind head on main.:exclamation: Current head 1143f1e differs from pull request most recent head 8d2fe73. Consider uploading reports for the commit 8d2fe73 to get more accurate results
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
It's going to be an immense pain to make the opus_id
Translator as complete as you've done for KAO, and I don't think it's necessary. The only reason we need that Translator is to convert between a "primary filespec" and an opus_id
. But the "primary filespec" is only taken from the index file. Thanks to Emilie's index file generator, we can control what files are listed in the index file, and we've decided that only a single resolution (the best) of each ring plane and the atmosphere will be included. This means we don't care about browse images, or counts-v-time
in the rings. All we care about is:
u0_kao_91cm_734nm_radius_<ring>_<ie>_100m.xml
global/u0_kao_91cm_734nm_radius_equator_<ie>_100m.xml
atmosphere/u0_kao_91cm_734nm_counts-v-time_atmos_<ie>.xml
Actually ignore a lot of what I said above...we'll discuss in the meeting today.
Status:
Here are some bundles that have missing opus id abbrev mapping in format doc or different detector values:
u137_hst_fos
hstfos-wfc3-occ-1996-076-u137
u138_hst_fos
hstfos-wfc3-occ-1996-101-u138
u144_caha_123cm
urn:nasa:pds:context:telescope:caha.1m23
but the format table has caha-calar_alto.1m23
instead of just caha.1m23
, not sure if we want the value to match just like other bundles.ir
as well in opus id, let me know if we want a different value like nicmos-ir
for this. Note: These two bundles has different detectors other than ir
and vis
gaas
ccd
For record, here is the start date (yod) for each bundle
{'u0201_palomar_508cm': ['2002-210'],
'u0_kao_91cm': ['1977-069'],
'u102a_irtf_320cm': ['1992-190'],
'u102b_irtf_320cm': ['1992-190'],
'u103_eso_220cm': ['1992-193'],
'u103_palomar_508cm': ['1992-193'],
'u1052_irtf_320cm': ['1988-133'],
'u11_ctio_400cm': ['1980-080'],
'u12_ctio_400cm': ['1980-229', '1980-228'],
'u12_eso_104cm': ['1980-229', '1980-228'],
'u12_eso_360cm': ['1980-229', '1980-228'],
'u12_lco_250cm': ['1980-229', '1980-228'],
'u134_saao_188cm': ['1995-252'],
'u137_hst_fos_540nm': ['1996-076'],
'u137_irtf_320cm': ['1996-076'],
'u138_hst_fos_540nm': ['1996-101'],
'u138_palomar_508cm': ['1996-101'],
'u13_sso_390cm': ['1981-116'],
'u144_caha_123cm': ['1997-273'],
'u144_saao_188cm': ['1997-273'],
'u149_irtf_320cm': ['1998-310'],
'u149_lowell_180cm': ['1998-310'],
'u14_ctio_150cm': ['1982-112'],
'u14_ctio_400cm': ['1982-112'],
'u14_eso_104cm': ['1982-112'],
'u14_lco_100cm': ['1982-112'],
'u14_lco_250cm': ['1982-112'],
'u14_opmt_106cm': ['1982-112'],
'u14_opmt_200cm': ['1982-112'],
'u14_teide_155cm': ['1982-112'],
'u15_mso_190cm': ['1982-121'],
'u16_palomar_508cm': ['1982-155'],
'u17b_saao_188cm': ['1983-084'],
'u23_ctio_400cm': ['1985-124'],
'u23_mcdonald_270cm': ['1985-124'],
'u23_teide_155cm': ['1985-124'],
'u25_ctio_400cm': ['1985-144'],
'u25_mcdonald_270cm': ['1985-144'],
'u25_palomar_508cm': ['1985-144'],
'u28_irtf_320cm': ['1986-116'],
'u2_teide_155cm': ['1977-357'],
'u34_irtf_320cm': ['1987-057'],
'u36_ctio_400cm': ['1987-092'],
'u36_irtf_320cm': ['1987-092', '1987-089'],
'u36_maunakea_380cm': ['1987-092', '1987-089'],
'u36_sso_230cm': ['1987-092'],
'u36_sso_390cm': ['1987-092'],
'u5_lco_250cm': ['1978-100'],
'u65_irtf_320cm': ['1990-172'],
'u83_irtf_320cm': ['1991-176'],
'u84_irtf_320cm': ['1991-179'],
'u9539_ctio_400cm': ['1993-181'],
'u9_lco_250cm': ['1979-161']}
hst-fos
. As part of adding this bundle to OPUS, I've added an HST FOS
instrument to go along with HST ACS
, etc. I'll add this to the OPUS ID documentation.caha.1m23
sounds like a problem with the bundle. That's not a valid telescope name. See the valid names in https://pds.nasa.gov/data/pds4/context-pds4/telescope/
You should talk to Mia and see if this came up when she was validating the bundle. We can certainly add this alias to the table (and to OPUS) to handle the bad bundle.hst-nicmos
? You shouldn't need an ir
. NICMOS is the detector.gaas
and `ccd.ir
or vis
". Aren't we using insb
for most of the generic IR detectors?Yes, as Rob said there’s an error in the LID reference, it should be urn:nasa:pds:context:telescope:caha-calar_alto.1m23
(and urn:nasa:pds:context:facility:observatory.caha-calar_alto
).
Note: there are both facility
and telescope
context products listed in collection_context.csv
. In the case of caha 123, there was an error in the telescope
LID (the facility
LID was valid), but in the case of Pic du Midi there was an error in the facility
LID (should be urn:nasa:pds:context:facility:observatory.pic_du_midi rather than urn:nasa:pds:context:facility:observatory.pic_du_midi-bigorre
) and the telescope LID was correct. Since I think you've been referring to the telescope
context products, the Pic du Midi facility
error didn’t appear.
I've been working on fixing these LID references (and some other issues) and updating the bundles so there will be another version coming up.
Thanks for the comment, I'll update them.
- I'm a little confused about the statement "other than
ir
orvis
". Aren't we usinginsb
for most of the generic IR detectors?
Sorry for the confusing comment, I meant to say vis
, ir
, and insb
. I forgot insb
, it's the most common one.
- HST FOS (Faint Object Spectrograph) has nothing to do with WFC3. It's a totally separate instrument. They didn't even exist at the same time. FOS was an original instrument, replaced by NICMOS during the 1997 servicing missing. WFC3 is a 4th generation instrument that was installed during the 2009 servicing mission. The OPUS ID prefix should be hst-fos. As part of adding this bundle to OPUS, I've added an HST FOS instrument to go along with HST ACS, etc. I'll add this to the OPUS ID documentation.
For these two hst-fos bundles, I was confused by the tag in the data label. They are something like this: (so I thought it was wfc3 abbrev)
<Observing_System_Component>
<name>Wide Field Camera 3</name>
<type>Instrument</type>
Also, just would like to point it out that they don't have telescope
in collection_context.csv
, not sure if this is expected.
Well I think you found another bug in the bundles. This is clearly wrong (especially since WFC3 didn't even exist in 1996):
<Observing_System_Component>
<name>Wide Field Camera 3</name>
<type>Instrument</type>
<Internal_Reference>
<lid_reference>urn:nasa:pds:context:instrument:hst.fos</lid_reference>
<reference_type>is_instrument</reference_type>
</Internal_Reference>
</Observing_System_Component>
Hopefully @mace-space can confirm and fix. For now, just assume the LID is correct and the text is wrong.
I don't think you need telescope
in the collection_context.csv
file because this observation didn't use a telescope...those are ground-based. It used HST, and they do have:
S,urn:nasa:pds:context:instrument:hst.fos
S,urn:nasa:pds:context:instrument_host:spacecraft.hst
Good spot on the bug for HST FOS – yes, those data labels should not say Wide Field Camera 3. I've added that to the list of things to fix, thanks.
Updates:
opus_id_to_primary_logical_path
opus_id_to_primary_logical_path
(make sure a pdsfile instance from from_opus_id
will return its corresponding primary filespec path when calling logical_path
). Pds4FileTest/pds4-holdings/bundles
Update:
opus_id_to_primary_logical_path
rule as wellos_path_exists
and os_path_isdir
to check filesystem before return False when SHELVES_ONLY
is set to True. This will make sure a pdsfile is cached as long as it exists in the filesystem. This is to fix from_path
when calling on an existing pdsfile with SHELVES_ONLY
set to True & missing shelves file.Update:
opus_type
opus_products
Pending Items:
Update opus_products rules:
*_atmos_index_*
or
*_atmosphere_index_*
in the file name.Update opus_products rules to
Comments updated by Dave on 04/05/24
Reference: Rob's email
OPUS Import and PdsFile rules for Uranus occultations
from 08/22/2023, 1:15PMDescription of changes:
opus_id
rules foruranus_occs_earthbased
prefix_mapping
, and use it to create rules for different opus idsopus_products
rules foruranus_occs_earthbased
with these changes different from reference email:*_atmos_index_*
or*_atmosphere_index_*
in the file name.opus_type
rules foruranus_occs_earthbased
opus_id_to_primary_logical_path
rules foruranus_occs_earthbased
opus_id
,opus_type
,opus_products
, andopus_id_to_primary_logical_path
test_opus_products
andtest_opus_id_to_primary_logical_path
inrules/uranus_occs_earthbased.py
test_opus_id
andtest_opus_type
intests/test_pds4file_blackbox.py
uranus_occs_earthbased_primary_filespec.py
and run this test for each file:from_logical_path
(path of a primary filespec)from_opus_id
holdings
tocls.PDS_HOLDINGS
inpdsfile.py
Know problems:
opus_type
rules