NeoGeographyToolkit / StereoPipeline

The NASA Ames Stereo Pipeline is a suite of automated geodesy & stereogrammetry tools designed for processing planetary imagery captured from orbiting and landed robotic explorers on other planets.
Apache License 2.0
478 stars 168 forks source link

Some distributed .dylib files are damaged preventing program use. #392

Closed rbeyer closed 10 months ago

rbeyer commented 1 year ago

Describe the bug I downloaded the latest release of ASP (3.2.0) onto my mac (12.6.3 Monterey, but still an Intel mac), unpacked the tarball, and tried to run pc_align. The OS threw up a series of dialogue boxes that indicated that libtbb.dylib and libGL.1.dylib were "damaged" and could not be opened. I tried with the 2023-02-01 daily and got the same behavior. While the dialogue boxes aren't helpful, the text sent to STDOUT has a little more info:

$> cd opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/bin/
$> ./pc_align -h
dyld[55738]: Library not loaded: '@rpath/libtbb.dylib'
  Referenced from: '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libembree.2.dylib'
  Reason: tried: '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' (code signature in <3837B61A-C21F-3524-864E-1DDAD65CA501> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libtbb.dylib' (code signature in <3837B61A-C21F-3524-864E-1DDAD65CA501> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/../lib/libtbb.dylib' (code signature in <3837B61A-C21F-3524-864E-1DDAD65CA501> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/../../../../../miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/oalexan1/miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libtbb.dylib' (code signature in <3837B61A-C21F-3524-864E-1DDAD65CA501> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/../lib/libtbb.dylib' (code signature in <3837B61A-C21F-3524-864E-1DDAD65CA501> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/../../../../../miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/oalexan1/miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libtbb.dylib' (code signature in <3837B61A-C21F-3524-864E-1DDAD65CA501> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/../lib/libtbb.dylib' (code signature in <3837B61A-C21F-3524-864E-1DDAD65CA501> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/../../../../../miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/oalexan1/miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libtbb.dylib' (code signature in <3837B61A-C21F-3524-864E-1DDAD65CA501> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libtbb.dylib' (code signature in <3837B61A-C21F-3524-864E-1DDAD65CA501> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/oalexan1/miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libtbb.dylib' (code signature in <3837B61A-C21F-3524-864E-1DDAD65CA501> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libtbb.dylib' (code signature in <3837B61A-C21F-3524-864E-1DDAD65CA501> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libtbb.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' (code signature in <3837B61A-C21F-3524-864E-1DDAD65CA501> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/3rdParty/lib/libtbb.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' (code signature in <3837B61A-C21F-3524-864E-1DDAD65CA501> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libtbb.dylib' not valid for use in process: library load disallowed by system policy)
Library not loaded: '@rpath/libGL.1.dylib'
  Referenced from: '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libisis7.1.0.dylib'
  Reason: tried: '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' (code signature in <C6005C01-EEFD-397A-8635-F8F580E1D80C> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libGL.1.dylib' (code signature in <C6005C01-EEFD-397A-8635-F8F580E1D80C> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/../lib/libGL.1.dylib' (code signature in <C6005C01-EEFD-397A-8635-F8F580E1D80C> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/../../../../../miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/oalexan1/miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libGL.1.dylib' (code signature in <C6005C01-EEFD-397A-8635-F8F580E1D80C> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/../lib/libGL.1.dylib' (code signature in <C6005C01-EEFD-397A-8635-F8F580E1D80C> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/../../../../../miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/oalexan1/miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libGL.1.dylib' (code signature in <C6005C01-EEFD-397A-8635-F8F580E1D80C> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/../lib/libGL.1.dylib' (code signature in <C6005C01-EEFD-397A-8635-F8F580E1D80C> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/../../../../../miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/oalexan1/miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libGL.1.dylib' (code signature in <C6005C01-EEFD-397A-8635-F8F580E1D80C> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libGL.1.dylib' (code signature in <C6005C01-EEFD-397A-8635-F8F580E1D80C> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/oalexan1/miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libGL.1.dylib' (code signature in <C6005C01-EEFD-397A-8635-F8F580E1D80C> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../lib/libGL.1.dylib' (code signature in <C6005C01-EEFD-397A-8635-F8F580E1D80C> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libGL.1.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' (code signature in <C6005C01-EEFD-397A-8635-F8F580E1D80C> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' not valid for use in process: library load disallowed by system policy), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/3rdParty/lib/libGL.1.dylib' (no such file), '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' (code signature in <C6005C01-EEFD-397A-8635-F8F580E1D80C> '/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/libGL.1.dylib' not valid for use in process: library load disallowed by system policy)

Unfortunately, both of these libraries look like dependencies that are related to ISIS, which always makes me nervous. Maybe we need to update our macOS build system?

Sadly, these errors means that I can't run ASP programs on my mac (good thing I also have access to a Linux system).

Looking at this huge block of text, the messages at first seem to indicate that maybe this is some mac gatekeeper nonsense ("not valid for use in process: library load disallowed by system policy"), but that's at odds with the "damaged" message.

However, farther down, there is this (carriage returns added):

'/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/libexec/../../../../../miniconda3/envs/isis7/lib/libtbb.dylib' (no such file),
'/Users/rbeyer/opt/StereoPipeline-3.2.1-alpha-2023-02-01-x86_64-OSX/lib/../../../../../miniconda3/envs/isis7/lib/libtbb.dylib' (no such file),
'/Users/oalexan1/miniconda3/envs/isis7/lib/libtbb.dylib' (no such file),

The rpath stuff has always mystified me in general, but does this mean that it is trying to find files way back up the directory chain in a very specific ISIS 7 miniconda3 directory, and that's the problem? And also Oleg's home directory path there looks weird. So is that something that didn't get cleansed properly in the build?

I suspect that there is something goofy with those two .dylib files in the distribution and that really is the problem, but when the system can't use those, it tries a variety of strategies to find alternate versions of those .dylib files which are functional.

If we can fix these libraries, great. Otherwise, we should (sadly) stop offering the macOS tarball and force people to install via conda I guess?

Your Environment (please complete the following information):

Additional context Add any other context about the problem here.

oleg-alexandrov commented 1 year ago

I will take a look. Does the conda install work for you? That is, without the tarball?

rbeyer commented 1 year ago

The 3.1.0 conda install (that I still have on my mac does), I haven't had the time to check the latest, but I'll get around to it.

rbeyer commented 1 year ago

Nope, conda install of 3.2.0 does not appear to work:

$> conda install stereo-pipeline==3.2.0
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

ResolvePackageNotFound:
  - llvm-openmp[version='15.0.6.*,>=15.0.6']

And it isn't clear what the remedy is.

oleg-alexandrov commented 1 year ago

That is odd. This package exists. See here: https://anaconda.org/conda-forge/llvm-openmp/15.0.6/download/osx-64/llvm-openmp-15.0.6-h61d9ccf_0.conda.

You can try to create a standalone env having this package only, to see if it really fails to get it. The command is:

conda create -n tmp_openmp llvm-openmp=15.0.6 -c conda-forge -y

Here's what I see:

The following NEW packages will be INSTALLED: llvm-openmp conda-forge/osx-64::llvm-openmp-15.0.6-h61d9ccf_0

If that works, you can try installing ASP using the precise env published at https://raw.githubusercontent.com/NeoGeographyToolkit/StereoPipeline/master/conda/asp_3.2.0_osx_env.yaml

You would need to fetch that, then, per the installation instructions in our doc, do:

conda env create -f asp_3.2.0_osx_env.yaml

As an alternative, this also worked for me:

conda create -n tmp_asp stereo-pipeline==3.2.0 -c nasa-ames-stereo-pipeline -c usgs-astrogeology -c conda-forge -y

In the meantime I reinstalled ISIS and will force the nightly to rebuild from a clean slate, and see how it goes. The hope is that it will fix the nightly build.

oleg-alexandrov commented 1 year ago

The latest build is up as well, after being given a dependency refresh and a wholesale rebuild. You can also try to fetch isis 7.1.0 in a stand-alone environment, to see if the problem stems from them.