Creating CSM ISDs

[oleg-alexandrov]

I am using the example at https://github.com/USGS-Astrogeology/knoten/blob/master/examples/mro_ctx_isis_cmp.ipynb

to create ISD json files. I fetched knoten and am using ISIS 4.4.0, the latest.

Things work, except for the fact that the lines:

kernels = ale.util.generate_kernels_from_cube(cub_loc, expand=True) usgscsm_str = ale.loads(fileName, props={'kernels': kernels}, formatter="usgscsm", verbose=False)

csm_isd = os.path.splitext(cub_loc)[0] + '.json' with open(csm_isd, 'w') as isd_file: isd_file.write(usgscsm_str)

creates .json files that are compatible with libusgscsm.so.1.4.0 and not with libusgscsm.so.1.5.1

That, despite my conda environment having

ale 0.8.5 py37h2527ec5_1 conda-forge csm 3.0.3.3 hc9558a2_0 conda-forge csmapi 1.0.1 py37h99015e2_0 conda-forge usgscsm 1.5.1 h4bd325d_0 conda-forge

Is the ale package able to create the latest CSM .json files?

Or maybe I am doing something wrong.

[jessemapel]

It looks like the notebooks haven't been updated since the switch to the new ISD format in USGSCSM 1.5. The fix for this is to remove the formatter argument in the ale.loads call.

The "usgscsm" and "isis" formatters are there for backwards compatibility, the default "ale" formatter is intended for use with USGSCM 1.5+ and ISIS 4.2+

[oleg-alexandrov]

Jesse, thank you.

It does not work though, either no formatter, or with the 'isis' or 'ale' formatter. I get errors like:

KeyError: 'IsisCube' KeyError: 'FILTER_NUMBER' Failed: 'INSTRUMENT_HOST_ID' Exception: No Such Driver for Label

I also tried things like:

from ale.formatters.formatter import to_isd

also with no luck.

I used the precise notebook, with ISIS 4.4. I tried ISIS 3.7 too, but after I synched up the ISIS data, I can no longer get ISIS 3.7 to work so I am not sure I did that experiment well with ISIS 3.7.

Any thoughts?

[jessemapel]

@oleg-alexandrov Can you set verbose=True on the load call to get debug output and post it?

[oleg-alexandrov]

If desired to reproduce this, likely it is best to create a fresh install of the latest dependencies with conda. (When doing that, I also got some error about pvl at some point, it seems that this package changed a function's name, but other packages that depend on it still seek the old name. )

[oleg-alexandrov]

Here it is is. Lots of repetitive stuff. It appears to do some kind of exhaustive search failing each time.

``` The grammar must be an instance of pvl.grammar. Trying Failed: 'IsisCube' Trying Failed: 'scipy.spatial.transform.rotation.Rotation' object has no attribute 'as_dcm' Trying Failed: 'CTX' Trying Failed: 'IsisCube' Trying Failed: None Trying Failed: 'NoneType' object is not subscriptable Trying Failed: 'IsisCube' Trying Failed: 'IsisCube' Trying Failed: 'CTX' Trying Failed: 'IsisCube' Trying Failed: 'INSTRUMENT_HOST_ID' Trying Failed: 'IsisCube' Trying Failed: 'IsisCube' Trying Failed: 'IsisCube' Trying Failed: Instrument ID is wrong. Trying Failed: Instrument ID is wrong. Trying Failed: 'FILTER_NUMBER' Trying Failed: 'IsisCube' Trying Failed: 'IsisCube' Trying Failed: 'IsisCube' Trying Failed: None Trying Failed: 'NoneType' object is not subscriptable Trying Failed: 'IsisCube' Trying Failed: 'IsisCube' Trying Failed: 'CTX' Trying Failed: 'IsisCube' Trying Failed: 'INSTRUMENT_HOST_ID' Trying Failed: 'IsisCube' Trying Failed: 'IsisCube' Trying Failed: 'IsisCube' Trying Failed: Instrument ID is wrong. Trying Failed: Instrument ID is wrong. Trying Failed: 'FILTER_NUMBER' Trying Failed: 'IsisCube' Trying Failed: 'IsisCube' Trying Failed: 'IsisCube' Trying Failed: 'IsisCube' Trying Failed: 'IsisCube' raceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/mro_drivers.py", line 83, in instrument_id return id_lookup[super().instrument_id] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 33, in instrument_id return self.label['IsisCube']['Instrument']['InstrumentId'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'IsisCube' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 119, in load isd = formatter(driver) File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/formatters/formatter.py", line 132, in to_isd instrument_pointing['constant_rotation'] = constant_rotation.rotation_matrix().flatten() File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/rotation.py", line 85, in rotation_matrix return self._rot.as_dcm() AttributeError: 'scipy.spatial.transform.rotation.Rotation' object has no attribute 'as_dcm' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/co_drivers.py", line 133, in instrument_id return id_lookup[super().instrument_id] KeyError: 'CTX' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/nh_drivers.py", line 35, in instrument_id return id_lookup[super().instrument_id] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 33, in instrument_id return self.label['IsisCube']['Instrument']['InstrumentId'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'IsisCube' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/selene_drivers.py", line 573, in instrument_id id = f"LISM_{self.base_band}" File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/selene_drivers.py", line 558, in base_band base_band = band_map[self.label.get("BASE_BAND")] base_band = band_map[self.label.get("BASE_BAND")] KeyError: None Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/selene_drivers.py", line 79, in instrument_id swath = self.label.get("SWATH_MODE_ID")[0] TypeError: 'NoneType' object is not subscriptable Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/viking_drivers.py", line 40, in instrument_id instrument_id = super().instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 33, in instrument_id return self.label['IsisCube']['Instrument']['InstrumentId'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'IsisCube' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/mess_drivers.py", line 288, in instrument_id return ID_LOOKUP[super().instrument_id] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 33, in instrument_id return self.label['IsisCube']['Instrument']['InstrumentId'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'IsisCube' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/mess_drivers.py", line 132, in instrument_id return ID_LOOKUP[super().instrument_id] KeyError: 'CTX' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/lro_drivers.py", line 298, in instrument_id return id_lookup[super().instrument_id] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 33, in instrument_id return self.label['IsisCube']['Instrument']['InstrumentId'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return self.label['IsisCube']['Instrument']['InstrumentId'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'IsisCube' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 119, in load isd = formatter(driver) File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/formatters/formatter.py", line 47, in to_isd start_lines, start_times, scan_rates = driver.line_scan_rate File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/type_sensor.py", line 38, in line_scan_rate t0_ephemeris = self.ephemeris_start_time - self.center_ephemeris_time File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/lro_drivers.py", line 162, in ephemeris_start_time start_time = spice.scs2e(self.spacecraft_id, self.label['LRO:SPACECRAFT_CLOCK_PREROLL_COUNT']) File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/data_naif.py", line 166, in spacecraft_id return spice.bods2c(self.spacecraft_name) File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/lro_drivers.py", line 57, in spacecraft_name return self.instrument_host_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_pds3.py", line 73, in instrument_host_id return self.label['INSTRUMENT_HOST_ID'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'INSTRUMENT_HOST_ID' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/lro_drivers.py", line 532, in instrument_id return id_lookup[super().instrument_id] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 33, in instrument_id return self.label['IsisCube']['Instrument']['InstrumentId'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'IsisCube' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/voyager_drivers.py", line 21, in instrument_id return sc_lookup[super().spacecraft_name] + '_' + sensor_lookup[super().instrument_id] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 57, in spacecraft_name return self.platform_name File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 46, in platform_name return self.label['IsisCube']['Instrument']['SpacecraftName'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 46, in platform_name return self.label['IsisCube']['Instrument']['SpacecraftName'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'IsisCube' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/mex_drivers.py", line 496, in instrument_id if(super().instrument_id != "HRSC"): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 33, in instrument_id return self.label['IsisCube']['Instrument']['InstrumentId'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'IsisCube' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/mex_drivers.py", line 171, in instrument_id raise Exception ("Instrument ID is wrong.") Exception: Instrument ID is wrong. Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/mex_drivers.py", line 649, in instrument_id raise Exception ("Instrument ID is wrong.") Exception: Instrument ID is wrong. Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/dawn_drivers.py", line 38, in instrument_id filter_number = self.filter_number File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_pds3.py", line 328, in filter_number return self.label['FILTER_NUMBER'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'FILTER_NUMBER' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/mro_drivers.py", line 29, in instrument_id return id_lookup[super().instrument_id] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/mro_drivers.py", line 29, in instrument_id return id_lookup[super().instrument_id] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 33, in instrument_id return self.label['IsisCube']['Instrument']['InstrumentId'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'IsisCube' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/co_drivers.py", line 349, in instrument_id return id_lookup[super().instrument_id] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 33, in instrument_id return self.label['IsisCube']['Instrument']['InstrumentId'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'IsisCube' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/isis_ideal_drivers.py", line 23, in instrument_id instrument_id = super().instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 33, in instrument_id return self.label['IsisCube']['Instrument']['InstrumentId'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'IsisCube' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/viking_drivers.py", line 134, in instrument_id instrument_id = super().instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 33, in instrument_id return self.label['IsisCube']['Instrument']['InstrumentId'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'IsisCube' Traceback (most recent call last): File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 116, in load res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/mess_drivers.py", line 73, in instrument_id res.instrument_id File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/mess_drivers.py", line 73, in instrument_id return ID_LOOKUP[super().instrument_id] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/base/label_isis.py", line 33, in instrument_id return self.label['IsisCube']['Instrument']['InstrumentId'] File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/pvl/collections.py", line 175, in __getitem__ return dict_getitem(self, key)[0] KeyError: 'IsisCube' Traceback (most recent call last): File "../eval_ctx.py", line 63, in verbose=True) File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 145, in loads res = load(label, props, formatter, verbose=verbose) File "/home/oalexan1/miniconda3/envs/knoten/lib/python3.7/site-packages/ale/drivers/__init__.py", line 128, in load raise Exception('No Such Driver for Label') Exception: No Such Driver for Label ```

[jessemapel]

Okay here's the issue:

Trying <class 'ale.drivers.mro_drivers.MroCtxPds3LabelNaifSpiceDriver'>
Failed: 'scipy.spatial.transform.rotation.Rotation' object has no attribute 'as_dcm'

scipy changed their rotation to/from rotation matrix methods in 1.6.0. This was fixed in https://github.com/USGS-Astrogeology/ale/pull/399 but hasn't been released yet. If you're working with ALE <= 0.8.5 then you can fix this by installing scipy < 1.6.0.

I'm going to close this issue as it will be fixed once ALE 0.8.6 comes out.