search

oscar-system / Oscar.jl

A comprehensive open source computer algebra system for computations in algebra, geometry, and number theory.
https://www.oscar-system.org
Other
339 stars 120 forks source link

polymake warning: couldn't run 4ti2: zsolve -q #691

Closed thofma closed 2 years ago

thofma commented 2 years ago

Seen on CI for macOS, julia 1.6.

polymake:  WARNING: rule _4ti2.hilbert, _4ti2.integer_points: HILBERT_BASIS_GENERATORS : CONE_AMBIENT_DIM , FACETS | INEQUALITIES failed: couldn't run 4ti2: zsolve -q /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T//poly11747Taaaa0001
Test Summary: | Pass  Total
Cone          |   20     20
Test Summary: | Pass  Total
Group         |    9      9
polymake:  WARNING: rule _4ti2.hilbert, _4ti2.integer_points: HILBERT_BASIS_GENERATORS : CONE_AMBIENT_DIM , FACETS | INEQUALITIES failed: couldn't run 4ti2: zsolve -q /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T//poly11747Taaaa0002

CC: @benlorenz

fingolfin commented 2 years ago

I've actually recently had some dialogs pop up by the macOS system crash reporter related to 4ti2. Unfortunately I am not sure when exactly it happened... when precompiling Oscar? running test suite? building the manual? It happened a few times (Julia 1.6.2 on macOS 10.14.6) but I ignored it so far as it was just randomly a few times. I actually thought it came from Singular.jl, though. If it ever happens again, I shall keep the crash log and look closer, perhaps it is related to the above?

fingolfin commented 2 years ago

OK, just happened when I tried to run the Oscar test suite. From my terminal:

...
Precompiling project...
  ✓ Oscar
  1 dependency successfully precompiled in 23 seconds (53 already precompiled)
  1 dependency precompiled but a different version is currently loaded. Restart julia to access the new version
     Testing Running tests...
sh: line 1: 92006 Abort trap: 6           zsolve -q /var/folders/d_/1zss2fnd6xdgclqnj8jj_27m0000gp/T//poly91999Taaaa0001 > /dev/null 2>&1
polymake:  WARNING: rule _4ti2.hilbert, _4ti2.integer_points: HILBERT_BASIS_GENERATORS : CONE_AMBIENT_DIM , FACETS | INEQUALITIES failed: couldn't run 4ti2: zsolve -q /var/folders/d_/1zss2fnd6xdgclqnj8jj_27m0000gp/T//poly91999Taaaa0001
Test Summary: | Pass  Total
Cone          |   20     20
Test Summary: | Pass  Total
Group         |    9      9
sh: line 1: 92028 Abort trap: 6           zsolve -q /var/folders/d_/1zss2fnd6xdgclqnj8jj_27m0000gp/T//poly91999Taaaa0002 > /dev/null 2>&1
polymake:  WARNING: rule _4ti2.hilbert, _4ti2.integer_points: HILBERT_BASIS_GENERATORS : CONE_AMBIENT_DIM , FACETS | INEQUALITIES failed: couldn't run 4ti2: zsolve -q /var/folders/d_/1zss2fnd6xdgclqnj8jj_27m0000gp/T//poly91999Taaaa0002
Test Summary: | Pass  Total
Polyhedron    |   29     29
Test Summary: | Pass  Total
PolyhedralFan |   17     17
sh: line 1: 92064 Abort trap: 6           zsolve -q /var/folders/d_/1zss2fnd6xdgclqnj8jj_27m0000gp/T//poly91999Taaaa0005 > /dev/null 2>&1
polymake:  WARNING: rule _4ti2.hilbert, _4ti2.integer_points: HILBERT_BASIS_GENERATORS : CONE_AMBIENT_DIM , FACETS | INEQUALITIES failed: couldn't run 4ti2: zsolve -q /var/folders/d_/1zss2fnd6xdgclqnj8jj_27m0000gp/T//poly91999Taaaa0005
Test Summary: | Pass  Total
OscarPolytope |   52     52
Test Summary: | Pass  Total
fmpz          |    9      9
Test Summary: | Pass  Total
fmpq          |   17     17
...

The crash reporter log is not very helpful, but at least slightly more info:

Process:               zsolve [92006]
Path:                  /Users/USER/*/zsolve
Identifier:            zsolve
Version:               0
Code Type:             X86-64 (Native)
Parent Process:        ??? [92005]
Responsible:           zsolve [92006]
User ID:               502

Date/Time:             2021-09-21 22:00:41.759 +0200
OS Version:            Mac OS X 10.14.6 (18G9323)
Report Version:        12

Time Awake Since Boot: 850000 seconds
Time Since Wake:       37000 seconds

System Integrity Protection: enabled

Crashed Thread:        0

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    DYLD, [0x1] Library missing

Application Specific Information:
dyld: launch, loading dependent libraries

Dyld Error Message:
  Library not loaded: @rpath/libgmpxx.4.dylib
  Referenced from: /Users/USER/*/zsolve
  Reason: image not found

Binary Images:
       0x10b938000 -        0x10b93fff7 +zsolve (0) <762A63D9-7390-300E-8DA7-1681FD380808> /Users/USER/*/zsolve
       0x10b944000 -        0x10b994ff7 +libzsolve.0.dylib (0) <116BB907-99B7-3370-A027-1DDB753093D0> /Users/USER/*/libzsolve.0.dylib
       0x10b9ad000 -        0x10b9b2fff +lib4ti2common.0.dylib (0) <E8A64D30-37D6-3225-A812-D249C72B06E9> /Users/USER/*/lib4ti2common.0.dylib
       0x10f9e6000 -        0x10fa5070f  dyld (655.1.1) <4842B61A-B868-3146-9CA9-5582C4198B25> /usr/lib/dyld

I guess the crucial bit is the dyld error about libgmpxx.4.dylib

benlorenz commented 2 years ago

This probably happens during some of the polyhedra tests but polymake falls back to a different code for the same computation, so this is not fatal. I will try have a look why the 4ti2 binaries / wrappers seem to fail on macos (this interface was not active before Polymake.jl 0.5.8).

benlorenz commented 2 years ago

@fingolfin As I don't have access to a mac right now, can you search for 4ti2_wrapper in your .julia/artifacts and in the same directory execute the circuits (or groebner) wrapper manually (the one for zsolve won't trigger any error unless you pass some input) and post the contents of the 4ti2_wrapper script please? Edit: I managed to extract the output from github actions and I am still trying to understand why it doesn't work on macos.

This script should set up the DYLD_FALLBACK_LIBRARY_PATH using the LIBPATH[] from the lib4ti2_jll in the same way JLLWrappers does it...

benlorenz commented 2 years ago

It seems the FileProducts from lib4ti2_jll don't really work on macOS. Plain lib4ti2_jll, note that I need to use exe4ti2int64() to set up the paths as groebner is just a FileProduct:

julia> lib4ti2_jll.exe4ti2int64() do cmd
       run(`$(lib4ti2_jll.exe4ti2int64_path) --version`)
       end
-------------------------------------------------
4ti2 version 1.6.9
Copyright 1998, 2002, 2006, 2015 4ti2 team.
4ti2 comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome
to redistribute it under certain conditions.
For details, see the file COPYING.
-------------------------------------------------
Using 64 bit integers.
Process(`/Users/lorenz/.julia/artifacts/4a603abc31013e04bcdcb12365beb6326a22acf9/bin/4ti2int64 --version`, ProcessExited(0))

julia> lib4ti2_jll.exe4ti2int64() do cmd
       run(`$(lib4ti2_jll.groebner) --version`)
       end
dyld: Library not loaded: @rpath/libglpk.40.dylib
  Referenced from: /Users/lorenz/.julia/artifacts/4a603abc31013e04bcdcb12365beb6326a22acf9/bin/4ti2int64
  Reason: image not found
/Users/lorenz/.julia/artifacts/4a603abc31013e04bcdcb12365beb6326a22acf9/bin/groebner: line 64:  9591 Abort trap: 6           "$DIR"/$EXECUTABLE $FUNCTION $@
ERROR: failed process: Process(`/Users/lorenz/.julia/artifacts/4a603abc31013e04bcdcb12365beb6326a22acf9/bin/groebner --version`, ProcessExited(134)) [134]

The same does work fine on Linux.

Edit: The above is a slightly different issue (but still a problem that should be fixed) as zsolve is in fact a proper binary and not just a script like groebner or circuits which we use in different places. I need to continue debugging the zolve issue later.

fingolfin commented 2 years ago

Just to understand: from where is zsolve called, exactly?

I know that we have some calls in Singular.jl, too, and back then I wrote regenerate_4ti2_wrapper() in https://github.com/oscar-system/Singular.jl/blob/master/src/setup.jl#L7 to create little shell script wrappers (those were/are necessary because the Singular library/kernel try to invoke these binaries; and using lib4ti2_jll.exe4ti2int64 do ... end did not really work (IIRC the reason is this: the actual zsolve "binary" is a shell script itself. So "executing" it actually executes a new shell process. But the shell is on a special list of "protected" system binaries, into which one cannot inject DYLD overrides from outside, for "security" reasons. But you can do it from inside (note that the shell wrapper I wrote use . to execute the script in the currently running shell, not in a new one).

So perhaps you need something similar in Polymake.jl, too?

benlorenz commented 2 years ago

polymake will look up zsolve in the path during init, the calls then happen somewhere deep in the polymake-perl code. To make this work I also prepare some wrappers that set the libpaths and then call the 4ti2 binaries, I did the wrappers slightly different by using export for the current shell process and using exec for the call (see here), which does work for zsolve (this is really a binary). But my approach does not work for groebner and circuits which are shell scripts, probably due to those security "features".

The reason that the Oscar.jl test complain is probably because polymake finds the zsolve symlink from Singular.jl (which does not set up any library paths).

So to summarize:

fingolfin commented 2 years ago

Perhaps we can use the same (or at least: identical) wrapper scripts for Polymake.jl and Singular.jl? If using . resp. source works, let's also insert a comment explaining the rationale. If we use identical setup, then this also solves the zsolve issue for you, right?

benlorenz commented 2 years ago

I have adjusted the wrappers in the new polymake_jll 400.500.0 version, but this still needs to propagate through the intermediate packages before it is available in Oscar. The code for the new wrappers is here: https://github.com/benlorenz/Yggdrasil/blob/098b6b56dd7390961f89f803cf203191c69f350c/P/polymake/bundled/patches/generate_deps_tree.jl#L62 It creates two wrapper scripts, one for the 4ti2 binaries using exec and a different one for the scripts using source. Then it will create symlinks for all available tools which can then be called by polymake or other software without setting any library paths. As usual these wrappers need to be regenerated during precompilation to make sure the paths are correct.

benlorenz commented 2 years ago

This should also be solved with the singular.jl release, and updates to polymake.jl.