cyrilbouvier commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues for a bug report that matches the one I want to file, without success.

Did you read the documentation and troubleshoot guide?

[X] I have read the documentation and troubleshoot guide

Environment

- **OS**: CentOS Linux release 7.3.1611 (Core)
- **Sage Version**: 9.8 (similar behaviour was observed with Sage 9.5 and Sage 9.6)

Steps To Reproduce

from a clean Sage 9.8 source directory

./configure make build ./sage sage: import sage.all

Config log

config.log

Package logs

No package failed to install.

Additional Information

The error is visible when trying to run ./sage:

$ ./sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.8, Release Date: 2023-02-11                     │
│ Using Python 3.9.13. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: sage.all is not available; this is a limited REPL.        ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: import sage.all
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In [1], line 1
----> 1 import sage.all

File /sage-9.8/src/sage/all.py:168
    166 from sage.modules.all    import *
    167 from sage.monoids.all    import *
--> 168 from sage.algebras.all   import *
    169 from sage.modular.all    import *
    170 from sage.sat.all        import *

File /sage-9.8/src/sage/algebras/all.py:21
      5 #*****************************************************************************
      6 #       Copyright (C) 2005 William Stein <wstein@gmail.com>
      7 #
   (...)
     17 #                  http://www.gnu.org/licenses/
     18 #*****************************************************************************
     19 from sage.misc.lazy_import import lazy_import
---> 21 import sage.algebras.catalog as algebras
     23 from .quatalg.all import *
     24 from .steenrod.all import *

File /sage-9.8/src/sage/algebras/catalog.py:84
      1 r"""
      2 Catalog of Algebras
      3 
   (...)
     80   <sage.algebras.yokonuma_hecke_algebra.YokonumaHeckeAlgebra>`
     81 """
     83 from sage.algebras.free_algebra import FreeAlgebra as Free
---> 84 from sage.algebras.quatalg.quaternion_algebra import QuaternionAlgebra as Quaternion
     85 from sage.algebras.steenrod.steenrod_algebra import SteenrodAlgebra as Steenrod
     86 from sage.algebras.finite_dimensional_algebras.finite_dimensional_algebra import FiniteDimensionalAlgebra as FiniteDimensional

File /sage-9.8/src/sage/algebras/quatalg/quaternion_algebra.py:68
     61 from operator import itemgetter
     63 from .quaternion_algebra_element import (
     64     QuaternionAlgebraElement_abstract,
     65     QuaternionAlgebraElement_generic,
     66     QuaternionAlgebraElement_rational_field,
     67     QuaternionAlgebraElement_number_field)
---> 68 from . import quaternion_algebra_cython
     70 from sage.modular.modsym.p1list import P1List
     72 from sage.misc.cachefunc import cached_method

File /sage-9.8/src/sage/algebras/quatalg/quaternion_algebra_cython.pyx:1, in init sage.algebras.quatalg.quaternion_algebra_cython()
----> 1 # distutils: language = c++
      2 # distutils: libraries = gmp m NTL_LIBRARIES
      3 # distutils: extra_compile_args = NTL_CFLAGS

File /sage-9.8/src/sage/matrix/matrix_integer_dense.pyx:1, in init sage.matrix.matrix_integer_dense()
----> 1 # -*- coding: utf-8 -*-
      2 # distutils: extra_compile_args = NTL_CFLAGS M4RI_CFLAGS
      3 # distutils: libraries = iml NTL_LIBRARIES gmp m CBLAS_LIBRARIES

ImportError: /lib64/libz.so.1: version `ZLIB_1.2.9' not found (required by /sage-9.8/local/lib/libpng16.so.16)

From what I understand from config.log, sage needed to install libz because the one on the machine was not new enough (it did not have function inflateValidate as required by build/pkgs/zlib/spkg-configure.m4):

$ nm -gD /lib64/libz.so | grep -i inflateV
#no output

It also needed to install libpng (even if it is available on the machine).

The error looks very similar to #31670, but the issue seems to indicate that it was fixed in 9.5.beta8. I try the same steps with sage 9.5 and sage 9.6 and it failed for both (the error message is the same but the failure happens directly when ./sage is run, without the need to do import all)

Here are some extract of objdump of the different libraries:

$ objdump -x local/lib/libpng16.so
...
Dynamic Section:
  NEEDED               libz.so.1
  NEEDED               libm.so.6
  NEEDED               libc.so.6
  SONAME               libpng16.so.16
  RPATH                /sage-9.8/local/lib
...
Version definitions:
1 0x01 0x08cf6c26 libpng16.so.16
2 0x00 0x052a4c70 PNG16_0

Version References:
  required from libm.so.6:
    0x09691a75 0x00 06 GLIBC_2.2.5
  required from libz.so.1:
    0x0827e5c9 0x00 05 ZLIB_1.2.9
    0x07e5ce34 0x00 04 ZLIB_1.2.3.4
  required from libc.so.6:
    0x06969194 0x00 07 GLIBC_2.14
    0x09691a75 0x00 03 GLIBC_2.2.5
...

$ objdump -x local/lib/libz.so
...
Dynamic Section:
  NEEDED               libc.so.6
  SONAME               libz.so.1
  RPATH                /sage-9.8/local/lib
...
Version definitions:
1 0x01 0x09d5f4e1 libz.so.1
2 0x00 0x0827e5c0 ZLIB_1.2.0
3 0x00 0x07e5cb32 ZLIB_1.2.0.2
    ZLIB_1.2.0
4 0x00 0x07e5cb38 ZLIB_1.2.0.8
    ZLIB_1.2.0.2
5 0x00 0x0827e5c2 ZLIB_1.2.2
    ZLIB_1.2.0.8
6 0x00 0x07e5cd33 ZLIB_1.2.2.3
    ZLIB_1.2.2
7 0x00 0x07e5cd34 ZLIB_1.2.2.4
    ZLIB_1.2.2.3
8 0x00 0x07e5ce33 ZLIB_1.2.3.3
    ZLIB_1.2.2.4
9 0x00 0x07e5ce34 ZLIB_1.2.3.4
    ZLIB_1.2.3.3
10 0x00 0x07e5ce35 ZLIB_1.2.3.5
    ZLIB_1.2.3.4
11 0x00 0x07e5d031 ZLIB_1.2.5.1
    ZLIB_1.2.3.5
12 0x00 0x07e5d032 ZLIB_1.2.5.2
    ZLIB_1.2.5.1
13 0x00 0x07e5c231 ZLIB_1.2.7.1
    ZLIB_1.2.5.2
14 0x00 0x0827e5c9 ZLIB_1.2.9
    ZLIB_1.2.7.1
...

$ objdump -x /lib64/libz.so
...
Dynamic Section:
  NEEDED               libc.so.6
  SONAME               libz.so.1
...
Version definitions:
1 0x01 0x09d5f4e1 libz.so.1
2 0x00 0x0827e5c0 ZLIB_1.2.0
3 0x00 0x07e5cb32 ZLIB_1.2.0.2
    ZLIB_1.2.0
4 0x00 0x07e5cb38 ZLIB_1.2.0.8
    ZLIB_1.2.0.2
5 0x00 0x0827e5c2 ZLIB_1.2.2
    ZLIB_1.2.0.8
6 0x00 0x07e5cd33 ZLIB_1.2.2.3
    ZLIB_1.2.2
7 0x00 0x07e5cd34 ZLIB_1.2.2.4
    ZLIB_1.2.2.3
8 0x00 0x07e5ce33 ZLIB_1.2.3.3
    ZLIB_1.2.2.4
9 0x00 0x07e5ce34 ZLIB_1.2.3.4
    ZLIB_1.2.3.3
10 0x00 0x07e5ce35 ZLIB_1.2.3.5
    ZLIB_1.2.3.4
11 0x00 0x07e5d031 ZLIB_1.2.5.1
    ZLIB_1.2.3.5
12 0x00 0x07e5d032 ZLIB_1.2.5.2
    ZLIB_1.2.5.1
...

$ ldd local/lib/libpng16.so 
    linux-vdso.so.1 =>  (0x00007ffff83b3000)
    libz.so.1 => /sage-9.8/local/lib/libz.so.1 (0x00007fd4f9234000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fd4f8f1a000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fd4f8b59000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fd4f967f000)

mkoeppe commented 1 year ago

Thanks for the detailed report. Could you try make -j list-broken-packages and post its output.

You could also try whether export LD_PRELOAD=/sage-9.8/local/lib/libz.so makes a difference

cyrilbouvier commented 1 year ago

Could you try make -j list-broken-packages and post its output.

See sage.bug.centos.log for the output

You could also try whether export LD_PRELOAD=/sage-9.8/local/lib/libz.so makes a difference

Running LD_PRELOAD=/sage-9.8/local/lib/libz.so ./sage works without errors: the message about the 'limited REPL' does not appear anymore, sage.all seems to be imported (for example, ZZ exists) and import sage.all run without errors. Thanks for the tip, I will use it like that for a while and see if I see encounter other problems.

cyrilbouvier commented 1 year ago

Even with LD_PRELOAD set up in my environment variables, I encounter a similar problem when trying to install the optional package cryptominisat: the ./configure script picked up the libz in /lib64/ and the make part ended in an error. Extract from logs/pkgs/cryptominisat-5.8.0.log

...
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.7")
...
g++  (...) /usr/lib64/libz.so (... more libs)
/src/sage-9.8/local/lib/libpng16.so.16: undefined reference to `inflateValidate@ZLIB_1.2.9'
collect2: error: ld returned 1 exit status

So I investigated a little bit more: I used LD_DEBUG=libs to list all libraries loaded when sage is run, and I use ldd on all of them to see which one had a dependency on libz. Most used the correct path /sage-9.8/src/local/lib but 7 of them were depending on libz from /lib64/:

/lib64/libcrypto.so.10
  ...
    libz.so.1 => /lib64/libz.so.1 (0x00007fcc30388000)
  ...

/lib64/libssl.so.10
  ...
    libz.so.1 => /lib64/libz.so.1 (0x00007f97a1f43000)
  ...

/sage-9.8/src/sage/symbolic/function.cpython-39-x86_64-linux-gnu.so
  ...
    libz.so.1 => /lib64/libz.so.1 (0x00007f1154513000)
  ...

/trinity/shared/apps/local/Python/3.9.13/lib/python3.9/lib-dynload/binascii.cpython-39-x86_64-linux-gnu.so
  ...
    libz.so.1 => /lib64/libz.so.1 (0x00007f5754554000)
  ...

/trinity/shared/apps/local/Python/3.9.13/lib/python3.9/lib-dynload/_hashlib.cpython-39-x86_64-linux-gnu.so
  ...
    libz.so.1 => /lib64/libz.so.1 (0x00007fe32858d000)
  ...

/trinity/shared/apps/local/Python/3.9.13/lib/python3.9/lib-dynload/_ssl.cpython-39-x86_64-linux-gnu.so
  ...
    libz.so.1 => /lib64/libz.so.1 (0x00007ffbc9616000)
  ...

/trinity/shared/apps/local/Python/3.9.13/lib/python3.9/lib-dynload/zlib.cpython-39-x86_64-linux-gnu.so
  ...
    libz.so.1 => /lib64/libz.so.1 (0x00007f69a9da9000)
  ...

It seems that my local version of python (which is used by Sage) depends on libz from /lib64. Could the problem come from here ?

Also from the 7 libraries, only one is compiled by sage (src/sage/symbolic/function.cpython-39-x86_64-linux-gnu.so). It has a RPATH with the correct value:

$ objdump -x src/sage/symbolic/function.cpython-39-x86_64-linux-gnu.so | grep RPATH
  RPATH                /sage-9.8/local/lib

but the output of ldd indicates that /lib64/libz.so.1 would be used. Could the problem come from here ?

mkoeppe commented 1 year ago

Thanks a lot for the additional info and investigation.

I agree that the problem is that Python builtin modules depend on the system zlib, which we reject.

(Not sure why our module sage.symbolic.function is the only Sage-built module with this dependency; this is rather strange.)

In Sage, we already reject system Python (and build our own) if some of its dependency libraries (bzip2 liblzma libffi) are not accepted from the system. It looks like we need to add zlib to this list. That's now #35638.

sagemath / sage

Centos 7.3: 'import sage.all' fails with 'ImportError: /lib64/libz.so.1: version `ZLIB_1.2.9' not found (required by /sage-9.8/local/lib/libpng16.so.16)' #35636

Is there an existing issue for this?

Did you read the documentation and troubleshoot guide?

Environment

Steps To Reproduce

from a clean Sage 9.8 source directory

Config log

Package logs

Additional Information