search

NeoGeographyToolkit / StereoPipeline

The NASA Ames Stereo Pipeline is a suite of automated geodesy & stereogrammetry tools designed for processing planetary imagery captured from orbiting and landed robotic explorers on other planets.
Apache License 2.0
479 stars 168 forks source link

stereo_parse failing to read camera models in Ubuntu 18.04 #275

Closed dbolan closed 2 years ago

dbolan commented 4 years ago

I know I emailed about this a while back, I think I've narrowed it down sufficiently for a bug report. Basically there is some package in the base Ubuntu 18.04 installation that breaks stereo_parse. The reproduction steps below are probably overkill because I was trying to prove to myself whether the problem was in ISIS or in ASP.

You will need:

Steps to reproduce:

None Traceback (most recent call last): File "/home/dan/asptest/asp262/libexec/stereo", line 124, in settings=run_and_parse_output( "stereo_parse", args, sep, opt.verbose ) File "/home/dan/asptest/asp262/libexec/asp_system_utils.py", line 264, in run_and_parse_output raise Exception('Failed executing: ' + " ".join(call)) Exception: Failed executing: /home/dan/asptest/asp262/libexec/stereo_parse l1.cub r1.cub output/results --threads 12 --corr-seed-mode 1

* If you run this `stereo_parse` command, you should see the following:

... Loading camera model: l1.cub

Error: ERROR Unable to initialize camera model from group [Instrument]. ERROR Unsupported camera model, unable to find plugin for SpacecraftName [LUNAR RECONNAISSANCE ORBITER] with InstrumentId [NACL]. ERROR Unable to find plugin [LroNarrowAngleCameraPlugin] in shared library [/home/dan/miniconda3/envs/isis/lib/LroNarrowAngleCamera].


* If you deactivate the conda environment for ISIS and set ISISROOT to point to the directory containing ASP, you should see the "Failed to convert string [1,79769313486232e+308] to a double" issue I've been talking about in a few other places.
* If you install updates with `sudo apt update; sudo apt upgrade`, you should see the same errors.
* If you run `stereo` with the cubes that worked on the bootable USB, you should see the same errors.
* Conversely, if you run `stereo` on from the bootable USB on the cubes that were made on the desktop, it should work.

**Comments**
Phew!
Even with all of this, I'm still not 100% sure that the problem lies in ASP, it just seems quite a bit more likely, since (as far as I've seen) current versions of ISIS don't store their camera model plugins like this anymore.
According to `cubediff` the cubes generated by ISIS are identical, though comparison is slightly complicated by the fact that my username on the bootable USB was 'ubuntu' instead of 'dan'. Poor foresight on my part, sorry. After a few sed queries changing '/home/dan' to '/home/ubuntu' and fixing the Bytes field in the History object to account for the changing string lengths of the label, `pvldiff` also reports no differences.
There are more ~200 more packages installed on the bootable USB stick than the minimal desktop installation. It's well past quitting time for me here, but I'll attach the diffs of the `dpkg -l` outputs on each system in case you want to take a look, and I'll see if I can find a chance to look at them tomorrow as well.

Thank you again for maintaining such a wonderful tool! I'm very excited to finally use it properly!

Attachment: [packages_diff.txt](https://github.com/NeoGeographyToolkit/StereoPipeline/files/3458071/packages_diff.txt)
dbolan commented 4 years ago

I was unable to reproduce this with Lubuntu 19.04, both on the bootable USB and on a proper installation. I'll probably just use ASP on Lubuntu for now, but considering the popularity of Ubuntu as an operating system I'd say this is still an issue.

Attaching all three package lists for posterity: packages-lubuntu-desktop.txt packages-ubuntu-desktop.txt packages-ubuntu-usb.txt

oleg-alexandrov commented 4 years ago

I believe this took you a lot of time. Sorry.

I finally built our code with ISIS 3.8, the latest. Maybe that was the problem. I went through the steps you mention, both on CentOS 7 and Ubuntu 18. Things work for me. I did not use the cubes with lronaccal applied to them, I think I was missing some calibration data and that command failed.

So here is one more attempt to straighten it out. You can try my build, at https://byss.arc.nasa.gov/oalexan1/StereoPipeline-x86_64-centos7.6.1810-2019-08-02_10-08-277997d78-dirty.tar.bz2

and maybe even with my own cubes (produced like you did) which are

https://byss.arc.nasa.gov/oalexan1/l0.cub

and

https://byss.arc.nasa.gov/oalexan1/r0.cub

and see how they compare with yours.

Note that I did not update my LRO NAC kernels for a while, but I don't think that is the problem. This is a software issue and not a data issue.

dbolan commented 4 years ago

Hi Oleg, thanks for giving this one more try. Unfortunately, this still doesn't work on a fresh installation. Tried with my own cubes and yours. Out of curiosity, were you able to reproduce this issue on your own machine? I found out a coworker ran into a similar issue (also on Ubuntu 18.04) some months ago, so I'm slightly more confident that it's not just an issue with my hardware or something -- though it working from the bootable USB would also make that possibility unlikely as well.

The main difference I'm seeing with your version running ISIS 3.8 is that I'm seeing Failed to convert string [1,79769313486232e+308] to a double before any other outputs. I've attached a copy of the full output because it's tricky to describe how it's different.

As for lronaccal, if you used the web service for spiceinit you may have hit this issue: USGS-Astrogeology/ISIS3/issues/1790

Thanks again for all your help.

oleg-alexandrov commented 4 years ago

No, I cannot reproduce the issue on my machine. I just tried it again with the build and cubes above on Ubuntu 18.04.2 LTS. I hope you did not change any paths and that it uses our own libraries. You can always edit the bin/stereo_pprc shell script and put there a line right before the last saying

ldd "${PROGRAM}"

to see if any libraries are picked from the wrong location. All libraries should be from our tarball except low level things like libc, libm, libX11, and just a few more.

As to that error about failing to convert a string, that is from ISIS indeed. The cub file itself has nothing so big as that number 1,79769313486232e+308, which is the largest double or close to it, the biggest value in the cub is the no-data value at -3.40282265508890445e+38.

Glad to hear at least Ubuntu 19 is working for you.

Since I can't reproduce it there is nothing I can do at this point.

oleg-alexandrov commented 4 years ago

Another user reported this, with HiRISE testcase ESP_042134_1985_RED.map.cub ESP_053962_1985_RED.map.cub, both on the Mac and Ubuntu 19 (referenced above). Hope to get to this when back from vacation in a few weeks and when not swamped by my other three projects at that time.

oleg-alexandrov commented 4 years ago

This issue came up again. This time I fetched the latest ISIS kernels and latest ISIS, and I still can't reproduce it. I ran things on Ubuntu 18.04. Adding my recipe here just for the record. It is odd that very few people run into this, but the issue occurs, and it is not clear what is going on.

Here's what I did with LRO NAC and it worked: Ensure the data is up to date:

cd path/to/your/isis3data;

rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isis3data/data/base . --exclude '*/kernels' rsync -azv --delete --partial isisdist.astrogeology.usgs.gov::isis3data/data/lro . --exclude '*/kernels'

Install ISIS with conda, per https://github.com/USGS-Astrogeology/ISIS3#installation Make a work directory. Go there. Do wget https://byss.arc.nasa.gov/oalexan1/l0.cub wget https://byss.arc.nasa.gov/oalexan1/r0.cub wget https://byss.arc.nasa.gov/stereopipeline/daily_build/StereoPipeline-2.6.2_post-2019-12-16-x86_64-Linux.tar.bz2 Untar ASP. export ISISROOT=/path/to/miniconda3/envs/isis3 export ISIS3DATA=path/to/your/isis3data env | grep ISIS # to see if variables were set correctly spiceinit from=l0.cub web = true spiceinit from=r0.cub web = true lronaccal from=l0.cub to=l0.cal.cub lronaccal from=r0.cub to=r0.cal.cub ./StereoPipeline-2.6.2_post-2019-12-16-x86_64-Linux/bin/stereo l0.cal.cub r0.cal.cub out/run

oleg-alexandrov commented 4 years ago

I was told that regional formats for numbers, dates and currency is the problem, with the specific example of the Italian format.

I guess the ISIS folks should watch for it too, though maybe it is us who don't do something right about the locale and then ISIS gets fed bad info. To be studied at some point.

dbolan commented 4 years ago

Thanks for keeping up on this! I do remember seeing this bug a while back which may a useful reference point. This particular instance was fixed in 3.9.0, but it seems very plausible that others are hanging around somewhere. If I run into this again I'll be sure to check locale.

rbeyer commented 4 years ago

Riccardo Pozzobon over on OpenPlanetary helped track down the issue with this error:

Error: ERROR Unable to initialize camera model from group [Instrument]. ERROR Failed to convert string [1,79769313486232e+308] to a double. Stereo step 0: Preprocessing failed

This may happen with using a binary version of ASP on a system with a locale setting that uses commas as the decimal separator. What happens is that you may have a newer version of ISIS that you use to prepare ISIS cubes before running stereo, and it uses your locale settings to write a number (somewhere, we still haven't tracked it down) as a string in your locale, like 1,79769313486232e+308, which is the largest value representable by a double. Then, when you run stereo, which uses a precompiled set of ISIS libraries, they try and read that string, and interprets it as 1.79769313486232e+322, which is larger than the allowable value representable by a double, and that leads to the error.

Essentially a version of ISIS that interprets written numbers one way tries to read in numbers written in a different way.

A work-around is to adjust the locale on your system, via editing your .bash_profile and/or .bashrc (or their equivalents on your system) to set the following environment variables (again, syntax may vary with your shell):

export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8

When you do this, and then do your ISIS pre-processing on your data, your shiny new ISIS will use those locale settings to write numbers with a period as the decimal separator, and then the version of ISIS baked into ASP (by us here at Ames where we were just using our native locale), will be able to read them, and you shouldn't have this error.

Thanks Ricardo!

oleg-alexandrov commented 1 year ago

Now this is done for the user in ASP itself. It will be in the daily build soon and later when ASP 3.1.1 is released.