search

DOI-USGS / ale

Abstraction Layer for Ephemerides (ALE)
Other
13 stars 32 forks source link

Running ALE on Mini-RF images #423

Closed thareUSGS closed 2 years ago

thareUSGS commented 3 years ago

I have never run ALE on a Mini-RF images and I don't think I have seen a Jupyter notebook or test data. I'm not sure what "state" (processing level or routines) the image needs to be pushed through.

I do have stereo images which are run through ISIS into an 8bit output cube here: /scratch/thare/SOCET_GXP/test_models_GXP/MINI-RF/Kingcrater/stereo_images/

there are previous version of the cubes here too: /archive/projects/lro/minirf/dems/KINGCRATER/ISIS/King_AHK/ and this file list the PDS IMG locations: /archive/projects/lro/minirf/dems/KINGCRATER/ISIS/King_AHK/get_images_King_final.sh

When I run on this LSZ_03204_1CD_XKU_04N120_S1.8bit.cub file I get this error:

Traceback (most recent call last):
  File "/scratch/thare/SOCET_GXP/test_models_GXP/MINI-RF/Kingcrater/stereo_images/run_ALE_cub.py", line 10, in <module>
    usgscsm_str = ale.loads(infile)
  File "/home/thare/.conda/envs/ale/lib/python3.9/site-packages/ale/drivers/__init__.py", line 156, in loads
    res = load(label, props, formatter, verbose=verbose)
  File "/home/thare/.conda/envs/ale/lib/python3.9/site-packages/ale/drivers/__init__.py", line 139, in load
    raise Exception('No Such Driver for Label')
Exception: No Such Driver for Label
jessemapel commented 3 years ago

Mini-RF is the only driver that still uses the old USGSCSM formatter. Try:

ale.load(label, formatter="usgscsm")
thareUSGS commented 3 years ago

no love - same error I tried ale.load and ale.loads. Do I need to extract the label from the Cube?

Traceback (most recent call last):
  File "/scratch/thare/SOCET_GXP/test_models_GXP/MINI-RF/Kingcrater/stereo_image                                                         s/run_ALE_cub.py", line 10, in <module>
    usgscsm_str = ale.loads(infile, formatter="usgscsm")
  File "/home/thare/.conda/envs/ale/lib/python3.9/site-packages/ale/drivers/__in                                                         it__.py", line 156, in loads
    res = load(label, props, formatter, verbose=verbose)
  File "/home/thare/.conda/envs/ale/lib/python3.9/site-packages/ale/drivers/__in                                                         it__.py", line 139, in load
    raise Exception('No Such Driver for Label')
Exception: No Such Driver for Label
jessemapel commented 3 years ago

@thareUSGS I was able to generate ISDs by setting the ALESPICEROOT environment variable to /scratch/spice which caused it to pick up the LRO meta kernels. The ISDs I got are lining up with previous testing when we originally developed this.

We do not have a driver in ALE that takes the SPICE data off of an ISIS cube, though. Is that something you would like?

thareUSGS commented 3 years ago

@jessemapel I am pretty sure I had the SPICE location ready. But I wonder if I only tested on cubes not original PDS images. I guess to be consistent it would probably be a good idea to grab from ISIS cube, but I am fine just support PDS as long as that is well documented or perhaps can be caught and warning issued (somehow).

jessemapel commented 3 years ago

I still ran ALE on the cube files to generate the ISDs, the driver just re-queries the spice kernels for ephemeris data instead of reading the tables off the cube.

jessemapel commented 3 years ago

@thareUSGS I have the generated ISDs here: /scratch/jmapel/minirf

thareUSGS commented 3 years ago

@jessemapel Thanks for building those I have them available in our tests wiki for usgscsm. Now for me to test ISD creation, should I build ALE or wait for a new release? I wasn't sure if you actually updated anything in ALE to get this to work.

jessemapel commented 3 years ago

I didn't update anything, all I did was set the ALESPICEROOT environment variable and use the usgscsm formatter. I can help you re-create them tomorrow if you'd like.

thareUSGS commented 3 years ago

Environment re-installed and seems to working now. However - almost working too well?

both lines worked: usgscsm_str = ale.loads(infile, formatter="usgscsm") and usgscsm_str = ale.loads(infile)

The later creating a ISD (*.json) about 3 time the size of the first line. I guess which one is recommended and is there a list I can maintain to know which style to run for what instrument (for now -- since we are still in flux). I can probably host at the Knoten main page or perhaps a wiki here.

jessemapel commented 3 years ago

They should both generate ISDs, but the sensor model isn't updated to work with the ale formatted one yet. So, you won't be able to get a sensor model with the second. Use

usgscsm_str = ale.loads(infile, formatter="usgscsm")

oleg-alexandrov commented 3 years ago

I am running into similar issues. I need the latest formatter to be able to use the latest released USGSCSM code ASP is hooked up to, presumably.

Will bringing the format up to speed entail just a bit of cosmetic work in places and maybe some more querying of spice data, or it a whole lot of new work? Once the new formatter is in place, will one be able to use the existing USGSCSM SAR camera model, or is there a lot of new work there as well?

If it is little work I could do it myself and make a pull request rather than waiting some months.

I am also getting an odd spiceinit error, it says "The camera is requesting spice data [BODY301_RADII] that was not attached" when I run it with ISIS 5.0.1 and with quite recent ISISDATA.

Oh, and spiceinit web = true gives me a segmentation fault, I noticed this with other data as well.

oleg-alexandrov commented 3 years ago

I was able to generate ISDs by setting the ALESPICEROOT environment variable to /scratch/spice which caused it to pick up the LRO meta kernels.

Would setting ALESPICEROOT to the same location as ISISROOT help? It did not work for me but maybe I missed something. Is it also expecting new kind of data beyond the usual kernels? Meaning, what is a "meta kernel"?

oleg-alexandrov commented 3 years ago

More spice issues with ALE when trying to use the driver LroMiniRfIsisLabelNaifSpiceDriver

I get the error: The first file '/usgs/cpkgs/isis3/data/lro/kernels/tspk/moon_pa_de421_1900-2050.bpc' specified by KERNELS_TO_LOAD in the file /home/oalexan1/projects/isis3data/lro/kernels/mk/lro_2018.tm could not be located.

That after fetching the latest ISISDATA for LRO, as rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isisdata/data/lro ~/projects/isis3data/

I don't know why the file I fetched, /home/oalexan1/projects/isis3data/lro/kernels/mk/lro_2018.tm, has absolute paths to some /usgs partition which should not be there.

I edited it, to make it point to my own ISISDATA, rather than to the USGS's local ISISDATA, but then a kernel is missing. I get the error:

The thirteenth file '/home/oalexan1/projects/isis3data/lro/kernels/sclk/lro_clkcor_2019198_v00.tsc' specified by KERNELS_TO_LOAD in the file /home/oalexan1/projects/isis3data/lro/kernels/mk/lro_2018.tm could not be located.

All I have is files like /home/oalexan1/projects/isis3data/lro/kernels/sclk/lro_clkcor_2021187_v00.tsc.

I tried to copy that file over to make SPICE happy, but then I got the error:

Insufficient ephemeris data has been loaded to compute the state of -85 (LUNAR RECONNAISSANCE ORBITER) relative to 0 (SOLAR SYSTEM BARYCENTER) at the ephemeris epoch 2010 DEC 19 00:17:24.815.

oleg-alexandrov commented 3 years ago

For the record, a good testcase here is lsz_03204_1cd_xku_06n120_v1.cub.

Amusingly enough, if a fresh .img is fetched from PDS, mrf2isis can get a cube, and spiceinit on the cube works, but the second spiceinit on the same cube does not work, and the kernels are also missing as before.

Oh, and the dataset lsz_01636_1cd_xku_09n120_v1 does have its kernels in ISISDATA, so I guess things fail just for some data.

oleg-alexandrov commented 3 years ago

I think this should be reopened, but I maybe several issues should be open instead describing all issues I mentioned earlier.

ryodohemmi commented 2 years ago

Any progress on this topic? I've still got the similar error when loading mini-RF img (w/ a spice path) and cube files (spiceinited) on the ALE 0.8.7.

oleg-alexandrov commented 2 years ago

Here are a couple more examples where creating Mini-RF CSM models based on spice kernels fails: lsz_03821_1cd_xku_16n196_v1.cub, lsz_03822_1cd_xku_23n196_v1.cub.

These belong to a notable acquisition where Mini-RF was meant to actually acquire stereo data, per Randy Kirk's publications, which is very rare for this sensor, so this is a valuable dataset.

jessemapel commented 2 years ago

I just downloaded lsz_03204_1cd_xku_06n120_v1 from here: https://pds-geosciences.wustl.edu/lro/lro-l-mrflro-4-cdr-v1/lromrf_0002/data/sar/03200_03299/level1/ ran it through mrf2isis and spiceinit and it worked without issue. Here's my kernels:

Group = Kernels
  NaifFrameCode             = -85700
  LeapSecond                = $base/kernels/lsk/naif0012.tls
  TargetAttitudeShape       = ($base/kernels/pck/pck00009.tpc,
                               $lro/kernels/pck/moon_080317.tf,
                               $lro/kernels/pck/moon_assoc_me.tf)
  TargetPosition            = ($lro/kernels/tspk/moon_pa_de421_1900-2050.bpc,
                               $lro/kernels/tspk/de421.bsp)
  InstrumentPointing        = ($lro/kernels/ck/moc42r_2010059_2010091_v04.bc,
                               $lro/kernels/fk/lro_frames_2014049_v01.tf)
  Instrument                = Null
  SpacecraftClock           = $lro/kernels/sclk/lro_clkcor_2022110_v00.tsc
  InstrumentPosition        = $lro/kernels/spk/fdf29r_2010060_2010091_v51.bsp
  InstrumentAddendum        = $lro/kernels/iak/mrflroAddendum002.ti
  ShapeModel                = $base/dems/ldem_128ppd_Mar2011_clon180_radius_p-
                              ad.cub
  InstrumentPositionQuality = Reconstructed
  InstrumentPointingQuality = Reconstructed
  CameraVersion             = 1
  Source                    = isis
End_Group

Try running through that but do not have ALESPICEROOT set.

oleg-alexandrov commented 2 years ago

I tried this. Without setting ALESPICEROOT I get the error: Failed: ale.spice_root is not set, cannot search for metakernels. ale.spice_root = "None". That's when it tries to use ale.drivers.lro_drivers.LroLrocPds3LabelNaifSpiceDriver.

Here's the python code:

import os, sys import json import ale

from ale import util

os.environ["ISISROOT"] = os.environ["HOME"] + "/miniconda3/envs/isis6" os.environ["ISISDATA"] = os.environ["HOME"] + "/projects/isis3data"

prefix = sys.argv[1]

if prefix.endswith(".cub") or prefix.endswith(".img") or prefix.endswith(".lbl"):

Wipe extension

prefix = os.path.splitext(prefix)[0]

cub_file = prefix + '.cub' print("Loading cub file: " + cub_file)

usgscsm_str = ale.loads(cub_file, formatter = "usgscsm", verbose = True)

csm_isd = os.path.splitext(cub_file)[0] + '.json' print("Saving " + csm_isd) with open(csm_isd, 'w') as isd_file: isd_file.write(usgscsm_str)

oleg-alexandrov commented 2 years ago

Here's the diff between my spiceinit result and yours:

\< SpacecraftClock = $lro/kernels/sclk/lro_clkcor_2021250_v00.tsc

> SpacecraftClock = $lro/kernels/sclk/lro_clkcor_2022110_v00.tsc 15c15,16 \< ShapeModel = Null

> ShapeModel = $base/dems/ldem_128ppd_Mar2011_clon180_radius_p- > ad.cub

Mine is on the left. I used shape = ellipsoid as suggested somewhere. Does the clock thing mean I need to update some kernels again?

ryodohemmi commented 2 years ago

Here's my result for lsz_03204_1cd_xku_06n120_v1:

$ wget https://pds-geosciences.wustl.edu/lro/lro-l-mrflro-4-cdr-v1/lromrf_0002/data/sar/03200_03299/level1/lsz_03204_1cd_xku_06n120_v1.img
$ wget https://pds-geosciences.wustl.edu/lro/lro-l-mrflro-4-cdr-v1/lromrf_0002/data/sar/03200_03299/level1/lsz_03204_1cd_xku_06n120_v1.lbl
$ wget https://pds-geosciences.wustl.edu/lro/lro-l-mrflro-4-cdr-v1/lromrf_0002/data/sar/03200_03299/level1/lsz_03204_1cd_xku_06n120_v1.txt

$ conda activate isis600
$ cat $ISISROOT/version
6.0.0 # Public version number
2021-08-31 # Release date
stable # Release stage (alpha, beta, stable)

$ mrf2isis from=lsz_03204_1cd_xku_06n120_v1.lbl to=lsz_03204_1cd_xku_06n120_v1.cub
$ spiceinit from=lsz_03204_1cd_xku_06n120_v1.cub spksmithed=true

Group = Kernels
  NaifFrameCode             = -85700
  LeapSecond                = $base/kernels/lsk/naif0012.tls
  TargetAttitudeShape       = ($base/kernels/pck/pck00009.tpc,
                               $lro/kernels/pck/moon_080317.tf,
                               $lro/kernels/pck/moon_assoc_me.tf)
  TargetPosition            = ($lro/kernels/tspk/moon_pa_de421_1900-2050.bpc,
                               $lro/kernels/tspk/de421.bsp)
  InstrumentPointing        = ($lro/kernels/ck/moc42r_2010059_2010091_v04.bc,
                               $lro/kernels/fk/lro_frames_2014049_v01.tf)
  Instrument                = Null
  SpacecraftClock           = $lro/kernels/sclk/lro_clkcor_2022089_v00.tsc
  InstrumentPosition        = $lro/kernels/spk/LRO_NO_06_201311_GRGM900C_L600-
                              .BSP
  InstrumentAddendum        = $lro/kernels/iak/mrflroAddendum002.ti
  ShapeModel                = $base/dems/ldem_128ppd_Mar2011_clon180_radius_p-
                              ad.cub
  InstrumentPositionQuality = Smithed
  InstrumentPointingQuality = Reconstructed
  CameraVersion             = 1
  Source                    = isis
End_Group

$ conda deactivate
$ conda activate ale

$ python
>>> import ale
>>> ale.__version__
'0.8.6'
>>> ale.loads('lsz_03204_1cd_xku_06n120_v1.cub')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 156, in loads
    res = load(label, props, formatter, verbose=verbose)
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 139, in load
    raise Exception('No Such Driver for Label')
Exception: No Such Driver for Label
jessemapel commented 2 years ago

It looks like the metakernel is messed up as reported in https://github.com/USGS-Astrogeology/ISIS3/issues/4930

Here's how to work around this:

import os, ale
test_cub = "lsz_03204_1cd_xku_06n120_v1.cub"
kernels = ale.util.generate_kernels_from_cube(test_cub, expand=True)
isd_str = ale.loads(test_cub, formatter="usgscsm", props={"kernels": kernels}, verbose=True)
isd_file = os.path.splitext(test_cub)[0] + '.json'
open(isd_file, 'w').write(isd_str)

The ale.util.generate_kernels_from_cube call reads the kernels used by spiceinit and passes them in. It requires that your cube be spiceinit'd first. I'm checking the LRO meta kernel right now.

Edit: I did this with ALE 0.8.5 shipped with ISIS 6.0.0

oleg-alexandrov commented 2 years ago

This works, thanks. But one has to use "loads" and not "load".

oleg-alexandrov commented 2 years ago

I hope this workaround is something temporary. I would guess generating kernels from cube should be the default, as this is consistent with how ISIS does things. In either case, the loader should be smart enough to try out whatever options are available rather than for the user to try out various things.

I also find it confusing that sometimes one has to set ALESPICEROOT and sometimes not. There's already ISISDATA, and that should be enough I think. Internally the software can do whatever setting and unsetting it may want but the user should not have to think about all this.

jessemapel commented 2 years ago

I agree it is very confusing and not a great solution. We're working on a project to replace all of the kernel selection logic that will address these issues. One of our big goals for the next fiscal year is to get everything properly aligned and fix some of the odd duplication that has cropped up from this work.

jessemapel commented 2 years ago

I'm also going to work on updating the merakernel we ship with the ISIS data area so that you can just use ALESPICEROOT, after you update the paths for your system.

oleg-alexandrov commented 2 years ago

That is good to know. That one has to update the paths on one's system is also not a great thing. I think if ISIS can process a cub file, then ALE should be able to as well, with no extra variables like ALESPICEROOT to set or extra work to do.

ryodohemmi commented 2 years ago

@jessemapel Thanks alot! That worked!

My motivation to use ale is the bundle adjustment (and later orthorecitification/stereo) of Mini-RF images via Ames Stereo Pipeline because the default ISIS3's jigsaw fails on a Mini-RF image.

Details When I ran it first, I got the error on ISISROOT:

kernels = ale.util.generate_kernels_from_cube(test_cub, expand=True)

/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/util.py:263: UserWarning: No IsisPreferences file found, is your ISISROOT env var set?
  warnings.warn("No IsisPreferences file found, is your ISISROOT env var set?")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/util.py", line 220, in generate_kernels_from_cube
    return get_kernels_from_isis_pvl(kernel_group, expand, format_as)
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/util.py", line 265, in get_kernels_from_isis_pvl
    kernels = [expandvars(expandvars(k, dict_to_lower(isisprefs['DataDirectory']))) for k in kernels]
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/util.py", line 265, in <listcomp>
    kernels = [expandvars(expandvars(k, dict_to_lower(isisprefs['DataDirectory']))) for k in kernels]
KeyError: 'DataDirectory'

So I re-ran it without conda deactivate (isis600) and conda activate ale with --stack, which gives no error. And ale.loads() output includes the first ~140 lines of the attempts and failures on applications of the other mission drivers like below. This is a bit unexpected to me.

$ conda activate --stack ale
$ python
>>> import os, ale
>>> test_cub = "lsz_03204_1cd_xku_06n120_v1.cub"
>>> kernels = ale.util.generate_kernels_from_cube(test_cub, expand=True)
>>> isd_str = ale.loads(test_cub, formatter="usgscsm", props={"kernels": kernels}, verbose=True)

Attempting to pre-parse label file
Successfully pre-parsed label file
Trying <class 'ale.drivers.voyager_drivers.VoyagerCameraIsisLabelNaifSpiceDriver'>
Failed: 'LUNAR RECONNAISSANCE ORBITER'

Traceback (most recent call last):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 127, in load
    res.instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/voyager_drivers.py", line 21, in instrument_id
    return sc_lookup[super().spacecraft_name] + '_' + sensor_lookup[super().instrument_id]
KeyError: 'LUNAR RECONNAISSANCE ORBITER'
Trying <class 'ale.drivers.viking_drivers.VikingIsisLabelNaifSpiceDriver'>
Failed: Instrument ID [MRFLRO] is wrong.

Traceback (most recent call last):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 127, in load
    res.instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/viking_drivers.py", line 43, in instrument_id
    raise Exception (f'Instrument ID [{instrument_id}] is wrong.')
Exception: Instrument ID [MRFLRO] is wrong.
Trying <class 'ale.drivers.mro_drivers.MroCtxIsisLabelNaifSpiceDriver'>
Failed: 'MRFLRO'

Traceback (most recent call last):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 127, in load
    res.instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/mro_drivers.py", line 83, in instrument_id
    return id_lookup[super().instrument_id]
KeyError: 'MRFLRO'
Trying <class 'ale.drivers.mro_drivers.MroCtxPds3LabelNaifSpiceDriver'>
Failed: 'INSTRUMENT_ID'

Traceback (most recent call last):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 127, in load
    res.instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/mro_drivers.py", line 194, in instrument_id
    return id_lookup[super().instrument_id]
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/base/label_pds3.py", line 33, in instrument_id
    return self.label['INSTRUMENT_ID']
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/pvl/collections.py", line 175, in __getitem__
    return dict_getitem(self, key)[0]
KeyError: 'INSTRUMENT_ID'
Trying <class 'ale.drivers.dawn_drivers.DawnFcPds3NaifSpiceDriver'>
Failed: 'INSTRUMENT_ID'

Traceback (most recent call last):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 127, in load
    res.instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/dawn_drivers.py", line 37, in instrument_id
    instrument_id = super().instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/base/label_pds3.py", line 33, in instrument_id
    return self.label['INSTRUMENT_ID']
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/pvl/collections.py", line 175, in __getitem__
    return dict_getitem(self, key)[0]
KeyError: 'INSTRUMENT_ID'
Trying <class 'ale.drivers.mex_drivers.MexHrscIsisLabelNaifSpiceDriver'>
Failed: Instrument ID is wrong.

Traceback (most recent call last):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 127, in load
    res.instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/mex_drivers.py", line 498, in instrument_id
    raise Exception ("Instrument ID is wrong.")
Exception: Instrument ID is wrong.
Trying <class 'ale.drivers.mex_drivers.MexHrscPds3NaifSpiceDriver'>
Failed: 'INSTRUMENT_ID'

Traceback (most recent call last):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 127, in load
    res.instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/mex_drivers.py", line 170, in instrument_id
    if(super().instrument_id != "HRSC"):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/base/label_pds3.py", line 33, in instrument_id
    return self.label['INSTRUMENT_ID']
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/pvl/collections.py", line 175, in __getitem__
    return dict_getitem(self, key)[0]
KeyError: 'INSTRUMENT_ID'
Trying <class 'ale.drivers.mex_drivers.MexSrcPds3NaifSpiceDriver'>
Failed: 'INSTRUMENT_ID'

Traceback (most recent call last):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 127, in load
    res.instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/mex_drivers.py", line 703, in instrument_id
    if(super().instrument_id != "HRSC"):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/base/label_pds3.py", line 33, in instrument_id
    return self.label['INSTRUMENT_ID']
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/pvl/collections.py", line 175, in __getitem__
    return dict_getitem(self, key)[0]
KeyError: 'INSTRUMENT_ID'
Trying <class 'ale.drivers.nh_drivers.NewHorizonsLorriIsisLabelNaifSpiceDriver'>
Failed: 'MRFLRO'

Traceback (most recent call last):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 127, in load
    res.instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/nh_drivers.py", line 35, in instrument_id
    return id_lookup[super().instrument_id]
KeyError: 'MRFLRO'
Trying <class 'ale.drivers.co_drivers.CassiniIssPds3LabelNaifSpiceDriver'>
Failed: 'INSTRUMENT_ID'

Traceback (most recent call last):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 127, in load
    res.instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/co_drivers.py", line 133, in instrument_id
    return id_lookup[super().instrument_id]
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/base/label_pds3.py", line 33, in instrument_id
    return self.label['INSTRUMENT_ID']
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/pvl/collections.py", line 175, in __getitem__
    return dict_getitem(self, key)[0]
KeyError: 'INSTRUMENT_ID'
Trying <class 'ale.drivers.lro_drivers.LroLrocIsisLabelNaifSpiceDriver'>
Failed: 'MRFLRO'

Traceback (most recent call last):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 127, in load
    res.instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/lro_drivers.py", line 300, in instrument_id
    return id_lookup[super().instrument_id]
KeyError: 'MRFLRO'
Trying <class 'ale.drivers.lro_drivers.LroLrocPds3LabelNaifSpiceDriver'>
Failed: 'INSTRUMENT_ID'

Traceback (most recent call last):
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/__init__.py", line 127, in load
    res.instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/drivers/lro_drivers.py", line 38, in instrument_id
    instrument = super().instrument_id
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/ale/base/label_pds3.py", line 33, in instrument_id
    return self.label['INSTRUMENT_ID']
  File "/home/hemmi/anaconda3/envs/ale/lib/python3.10/site-packages/pvl/collections.py", line 175, in __getitem__
    return dict_getitem(self, key)[0]
KeyError: 'INSTRUMENT_ID'
Trying <class 'ale.drivers.lro_drivers.LroMiniRfIsisLabelNaifSpiceDriver'>
Success with:  <ale.drivers.lro_drivers.LroMiniRfIsisLabelNaifSpiceDriver object at 0x7f1cefc494e0>
ISD:
 {
  "image_lines": 4877,
  "image_samples": 2410,
  "name_platform": "LUNAR RECONNAISSANCE ORBITER",
  "name_sensor": "MINI-RF LRO",
  "radii": {
    "semimajor": 1737.4,
    "semiminor": 1737.4,
    "unit": "km"
  },
...
*The rest is omitted
jessemapel commented 2 years ago

That's just the verbose output you're seeing. It successfully generated an ISD for you see:

Success with:  <ale.drivers.lro_drivers.LroMiniRfIsisLabelNaifSpiceDriver object at 0x7f1cefc494e0>

You should be able to bundle adjust mini-RF images with jigsaw, I'm curious what failure you're seeing.

oleg-alexandrov commented 2 years ago

My motivation to use ale is the bundle adjustment (and later orthorecitification/stereo) of Mini-RF images via Ames Stereo Pipeline because the default ISIS3's jigsaw fails on a Mini-RF image.

I am flattered you want to use ASP. But note that MiniRF is a tough nut. See https://github.com/USGS-Astrogeology/usgscsm/issues/369 and https://github.com/USGS-Astrogeology/usgscsm/issues/372. Bundle adjustment may still fail for you because interest points are hard to find (and I just fixed a bug in ASP, it was choking on multi-band cubes as what exists for MiniRF).

If you run into issues with ASP you can file an issue at ASP's GitHub repo and let me know your dataset.

So I re-ran it without conda deactivate (isis600) and conda activate ale with --stack, which gives no error. And ale.loads() output includes the first ~140 lines of the attempts and failures on applications of the other mission drivers like below. This is a bit unexpected to me.

Use

isd_str = ale.loads(test_cub, formatter="usgscsm", props={"kernels": kernels}, verbose=False)

Note that the verbose flag was set to False.

Long-term, trying all possible drivers and failing is slow. There should be some lookup table for which drivers to try, based on metadata from the .cub file.

ryodohemmi commented 2 years ago

Thank you again. Setting verbose=False can hide the messages.

Running jigsaw (ISIS v6.0.0) on a single mini-RF cube fails (says 'Segmentation fault (core dumped)') when setting camsolve=angles|velocities|accelerations|all.

$ jigsaw fromlist=04425.s1.NoZ.lis cnet=04425.net onet=04425jigsaw.net  \
  observations=yes maxits=100 \
  camsolve=angles spsolve=none spacecraft_position_sigma=300 \
  spacecraft_velocity_sigma=1 camera_angles_sigma=0.1 camera_angular_velocity_sigma=0.1 \
  update=no

Validating network...
Validation complete!...
starting iteration 1

Segmentation fault (core dumped)

04425.net contains twelve Constrained points that were manually taken using qnet with an orthomosaic and a DEM. Jigsaw can work only when setting camsolve=none (regardless of spsolve options). It converges but its result (a qualitative comparison b/w a jigsaw'ed cube and an orthomosaic at the same location using qnet) looks not very good (i.e., xy offsets relative to the reference are not greatly reduced).

$ jigsaw fromlist=04425.s1.NoZ.lis cnet=04425.net onet=04425jigsaw.net \
  observations=yes maxits=100 \
  camsolve=none spsolve=all spacecraft_position_sigma=300 \
  spacecraft_velocity_sigma=1 camera_angles_sigma=0.1 camera_angular_velocity_sigma=0.1 \
  update=no

Validating network...
Validation complete!...
starting iteration 1

Iteration: 1 
Sigma0:         0.0307660459 
Observations: 24 
Constrained Parameters:36 
Unknowns: 45 
Degrees of Freedom: 21 
End of Iteration 1 
Elapsed Time:         0.0022610000 
Group = Iteration1
  Sigma0                       = 0.03076604593449
  Observations                 = 24
  Constrained_Point_Parameters = 36
  Constrained_Image_Parameters = 6
  Unknown_Parameters           = 45
  Degrees_of_Freedom           = 21
  Rejected_Measures            = 0
End_Group
starting iteration 2

Iteration: 2 
Sigma0:         0.0307612873 
Observations: 24 
Constrained Parameters:36 
Unknowns: 45 
Degrees of Freedom: 21 
End of Iteration 2 
Elapsed Time:         0.0020490000 

Group = Iteration2
  Sigma0                       = 0.030761287321988
  Observations                 = 24
  Constrained_Point_Parameters = 36
  Constrained_Image_Parameters = 6
  Unknown_Parameters           = 45
  Degrees_of_Freedom           = 21
  Rejected_Measures            = 0
End_Group
starting iteration 3

Iteration: 3 
Sigma0:         0.0307612871 
Observations: 24 
Constrained Parameters:36 
Unknowns: 45 
Degrees of Freedom: 21 
End of Iteration 3 
Elapsed Time:         0.0019920000 

Group = Iteration3
  Sigma0                       = 0.030761287068864
  Observations                 = 24
  Constrained_Point_Parameters = 36
  Constrained_Image_Parameters = 6
  Unknown_Parameters           = 45
  Degrees_of_Freedom           = 21
  Rejected_Measures            = 0
End_Group
starting iteration 4

Iteration: 4 
Sigma0:         0.0307612871 
Observations: 24 
Constrained Parameters:36 
Unknowns: 45 
Degrees of Freedom: 21 
Bundle has converged

Bundle Complete

Group = "Iteration4: Final"
  Sigma0                       = 0.030761287060391
  Observations                 = 24
  Constrained_Point_Parameters = 36
  Constrained_Image_Parameters = 6
  Unknown_Parameters           = 45
  Degrees_of_Freedom           = 21
  Rejected_Measures            = 0
  Converged                    = TRUE
  TotalElapsedTime             = 0.013111
End_Group

Generating report files

Group = Iteration1
  Sigma0                       = 0.03076604593449
  Observations                 = 24
  Constrained_Point_Parameters = 36
  Constrained_Image_Parameters = 6
  Unknown_Parameters           = 45
  Degrees_of_Freedom           = 21
  Rejected_Measures            = 0
End_Group

Group = Iteration2
  Sigma0                       = 0.030761287321988
  Observations                 = 24
  Constrained_Point_Parameters = 36
  Constrained_Image_Parameters = 6
  Unknown_Parameters           = 45
  Degrees_of_Freedom           = 21
  Rejected_Measures            = 0
End_Group

Group = Iteration3
  Sigma0                       = 0.030761287068864
  Observations                 = 24
  Constrained_Point_Parameters = 36
  Constrained_Image_Parameters = 6
  Unknown_Parameters           = 45
  Degrees_of_Freedom           = 21
  Rejected_Measures            = 0
End_Group

Group = "Iteration4: Final"
  Sigma0                       = 0.030761287060391
  Observations                 = 24
  Constrained_Point_Parameters = 36
  Constrained_Image_Parameters = 6
  Unknown_Parameters           = 45
  Degrees_of_Freedom           = 21
  Rejected_Measures            = 0
  Converged                    = TRUE
  TotalElapsedTime             = 0.013111
End_Group

Group = JigsawResults
  Status = "Camera pointing NOT updated"
End_Group
jessemapel commented 2 years ago

You do not want to solve for any pointing with radar sensors because there is no pointing involved, it's just a range from the sensor.

Try redoing your solution but setting OVERHERMITE=TRUE. This will solve for a bias instead of replacing the original position. This tends to work better for long images because it preserves the original "shape" of the ephemerides.

ryodohemmi commented 2 years ago

@jessemapel Thanks for giving a tip about jigsaw I overlooked! Information on appropriate mini-RF data handling is very limited online so it is very helpful to me. I tried running jigsaw many many times all day long and found the undesired misalignments come from jigsaw's sigma values I set. Finally I got the result I want by simply deleting all the sigma values.

jigsaw fromlist=04425.s1.NoZ.lis cnet=04425.net \
       file_prefix=04225 onet=04425_jigsaw.net \
       observations=no radius=yes maxits=100 \
       camsolve=none spsolve=all \
       overhermite=yes \
       update=no

Its sigma0 value was improved 10 times smaller than previous one. 

Group = "Iteration4: Final"
  Sigma0                       = 0.0046509224009294
  Observations                 = 34
  Constrained_Point_Parameters = 51
  Constrained_Image_Parameters = 0
  Unknown_Parameters           = 60
  Degrees_of_Freedom           = 25
  Rejected_Measures            = 0
  Converged                    = TRUE
  TotalElapsedTime             = 0.01758
End_Group
jessemapel commented 2 years ago

Done as we've confirmed that ALE is working correctly with LRO Mini-RF data now. It's still off by a handful of pixels from ISIS, but it's not egregiously incorrect.