search

facebookresearch / nle

The NetHack Learning Environment
Other
940 stars 114 forks source link

Can't install on Linux #297

Closed maximzxc closed 2 years ago

maximzxc commented 2 years ago

I'm getting the following error on pip install nle

Collecting nle
  Using cached nle-0.8.1.tar.gz (6.9 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: pybind11>=2.2 in /home/maximzxc/miniconda3/envs/nle/lib/python3.8/site-packages (from nle) (2.8.1)
Requirement already satisfied: numpy>=1.16 in /home/maximzxc/miniconda3/envs/nle/lib/python3.8/site-packages (from nle) (1.21.5)
Requirement already satisfied: gym>=0.15 in /home/maximzxc/miniconda3/envs/nle/lib/python3.8/site-packages (from nle) (0.21.0)
Requirement already satisfied: cloudpickle>=1.2.0 in /home/maximzxc/miniconda3/envs/nle/lib/python3.8/site-packages (from gym>=0.15->nle) (2.0.0)
Building wheels for collected packages: nle
  Building wheel for nle (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/maximzxc/miniconda3/envs/nle/bin/python /home/maximzxc/miniconda3/envs/nle/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp1b6d874n
       cwd: /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0
  Complete output (180 lines):
  fatal: not a git repository (or any of the parent directories): .git
  Building wheel nle-0.8.1
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/nle
  copying nle/version.py -> build/lib.linux-x86_64-3.8/nle
  copying nle/__init__.py -> build/lib.linux-x86_64-3.8/nle
  creating build/lib.linux-x86_64-3.8/nle/env
  copying nle/env/__init__.py -> build/lib.linux-x86_64-3.8/nle/env
  copying nle/env/base.py -> build/lib.linux-x86_64-3.8/nle/env
  copying nle/env/tasks.py -> build/lib.linux-x86_64-3.8/nle/env
  creating build/lib.linux-x86_64-3.8/nle/nethack
  copying nle/nethack/__init__.py -> build/lib.linux-x86_64-3.8/nle/nethack
  copying nle/nethack/nethack.py -> build/lib.linux-x86_64-3.8/nle/nethack
  copying nle/nethack/actions.py -> build/lib.linux-x86_64-3.8/nle/nethack
  creating build/lib.linux-x86_64-3.8/nle/agent
  copying nle/agent/__init__.py -> build/lib.linux-x86_64-3.8/nle/agent
  copying nle/agent/agent.py -> build/lib.linux-x86_64-3.8/nle/agent
  copying nle/agent/vtrace.py -> build/lib.linux-x86_64-3.8/nle/agent
  creating build/lib.linux-x86_64-3.8/nle/scripts
  copying nle/scripts/ttyrec.py -> build/lib.linux-x86_64-3.8/nle/scripts
  copying nle/scripts/read_heaplog.py -> build/lib.linux-x86_64-3.8/nle/scripts
  copying nle/scripts/check_nethack_speed.py -> build/lib.linux-x86_64-3.8/nle/scripts
  copying nle/scripts/__init__.py -> build/lib.linux-x86_64-3.8/nle/scripts
  copying nle/scripts/read_tty.py -> build/lib.linux-x86_64-3.8/nle/scripts
  copying nle/scripts/play.py -> build/lib.linux-x86_64-3.8/nle/scripts
  copying nle/scripts/collect_env.py -> build/lib.linux-x86_64-3.8/nle/scripts
  copying nle/scripts/ttyplay.py -> build/lib.linux-x86_64-3.8/nle/scripts
  copying nle/scripts/test_raw_nethack.py -> build/lib.linux-x86_64-3.8/nle/scripts
  copying nle/scripts/ttyplay2.py -> build/lib.linux-x86_64-3.8/nle/scripts
  copying nle/scripts/plot.py -> build/lib.linux-x86_64-3.8/nle/scripts
  package init file 'nle/tests/__init__.py' not found (or not a regular file)
  creating build/lib.linux-x86_64-3.8/nle/tests
  copying nle/tests/test_system.py -> build/lib.linux-x86_64-3.8/nle/tests
  copying nle/tests/test_profile.py -> build/lib.linux-x86_64-3.8/nle/tests
  copying nle/tests/test_nethack.py -> build/lib.linux-x86_64-3.8/nle/tests
  copying nle/tests/test_envs.py -> build/lib.linux-x86_64-3.8/nle/tests
  running build_ext
  -- The C compiler identification is GNU 11.2.0
  -- The CXX compiler identification is GNU 11.2.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /usr/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  Release build.
  -- Building nle backend version: 0.8.1
  -- HACKDIR set to: /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/build/lib.linux-x86_64-3.8/nle/nethackdir
  Seeding enabled.
  -- The ASM compiler identification is GNU
  -- Found assembler: /usr/bin/cc
  -- pybind11 v2.6.2
  CMake Warning (dev) at /usr/share/cmake-3.22/Modules/CMakeDependentOption.cmake:84 (message):
    Policy CMP0127 is not set: cmake_dependent_option() supports full Condition
    Syntax.  Run "cmake --help-policy CMP0127" for policy details.  Use the
    cmake_policy command to set the policy and suppress this warning.
  Call Stack (most recent call first):
    third_party/pybind11/CMakeLists.txt:98 (cmake_dependent_option)
  This warning is for project developers.  Use -Wno-dev to suppress it.

  -- Found PythonInterp: /home/maximzxc/miniconda3/envs/nle/bin/python (found version "3.8.12")
  -- Found PythonLibs: /home/maximzxc/miniconda3/envs/nle/lib
  -- Performing Test HAS_FLTO
  -- Performing Test HAS_FLTO - Success
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/build/temp.linux-x86_64-3.8
  Scanning dependencies of target fcontext
  [  1%] Building ASM object third_party/deboost.context/CMakeFiles/fcontext.dir/asm/make_x86_64_sysv_elf_gas.S.o
  [  1%] Building C object util/CMakeFiles/dlb.dir/dlb_main.c.o
  [  1%] Building C object util/CMakeFiles/recover.dir/recover.c.o
  [  1%] Building C object util/CMakeFiles/dlb.dir/__/src/dlb.c.o
  [  1%] Building C object util/CMakeFiles/makedefs.dir/makedefs.c.o
  [  1%] Building C object util/CMakeFiles/makedefs.dir/__/src/objects.c.o
  [  4%] Building C object util/CMakeFiles/makedefs.dir/__/src/monst.c.o
  [  6%] Building ASM object third_party/deboost.context/CMakeFiles/fcontext.dir/asm/ontop_x86_64_sysv_elf_gas.S.o
  [  6%] Building C object third_party/deboost.context/CMakeFiles/fcontext.dir/source/stack.c.o
  [  1%] Building C object util/CMakeFiles/dlb.dir/panic.c.o
  [  6%] Building C object util/CMakeFiles/dlb.dir/__/src/alloc.c.o
  [  6%] Building C object CMakeFiles/nethackdl.dir/sys/unix/nledl.c.o
  [  6%] Building ASM object third_party/deboost.context/CMakeFiles/fcontext.dir/asm/jump_x86_64_sysv_elf_gas.S.o
  In file included from /usr/include/features.h:484,
                   from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                   from /usr/include/stdio.h:27,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/global.h:9,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/config.h:594,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/src/alloc.c:11:
  /usr/include/x86_64-linux-gnu/sys/cdefs.h:345:73: error: macro "__has_attribute" requires an identifier
    345 | #if __GNUC_PREREQ (3,4) || __glibc_has_attribute (__warn_unused_result__)
        |                                                                         ^
  In file included from /usr/include/features.h:484,
                   from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                   from /usr/include/stdio.h:27,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/global.h:9,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/config.h:594,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/util/dlb_main.c:8:
  /usr/include/x86_64-linux-gnu/sys/cdefs.h:345:73: error: macro "__has_attribute" requires an identifier
    345 | #if __GNUC_PREREQ (3,4) || __glibc_has_attribute (__warn_unused_result__)
        |                                                                         ^
  In file included from /usr/include/features.h:484,
                   from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                   from /usr/include/stdio.h:27,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/global.h:9,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/config.h:594,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/src/monst.c:6:
  /usr/include/x86_64-linux-gnu/sys/cdefs.h:345:73: error: macro "__has_attribute" requires an identifier
    345 | #if __GNUC_PREREQ (3,4) || __glibc_has_attribute (__warn_unused_result__)
        |                                                                         ^
  In file included from /usr/include/features.h:484,
                   from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                   from /usr/include/stdio.h:27,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/global.h:9,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/config.h:594,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/util/makedefs.c:11:
  /usr/include/x86_64-linux-gnu/sys/cdefs.h:345:73: error: macro "__has_attribute" requires an identifier
    345 | #if __GNUC_PREREQ (3,4) || __glibc_has_attribute (__warn_unused_result__)
        |                                                                         ^
  In file included from /usr/include/features.h:484,
                   from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                   from /usr/include/stdio.h:27,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/global.h:9,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/config.h:594,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/util/panic.c:12:
  /usr/include/x86_64-linux-gnu/sys/cdefs.h:345:73: error: macro "__has_attribute" requires an identifier
    345 | #if __GNUC_PREREQ (3,4) || __glibc_has_attribute (__warn_unused_result__)
        |                                                                         ^
  In file included from /usr/include/features.h:484,
                   from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                   from /usr/include/stdio.h:27,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/global.h:9,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/config.h:594,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/src/objects.c:35:
  /usr/include/x86_64-linux-gnu/sys/cdefs.h:345:73: error: macro "__has_attribute" requires an identifier
    345 | #if __GNUC_PREREQ (3,4) || __glibc_has_attribute (__warn_unused_result__)
        |                                                                         ^
  In file included from /usr/include/features.h:484,
                   from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                   from /usr/include/stdio.h:27,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/global.h:9,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/config.h:594,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/util/recover.c:15:
  /usr/include/x86_64-linux-gnu/sys/cdefs.h:345:73: error: macro "__has_attribute" requires an identifier
    345 | #if __GNUC_PREREQ (3,4) || __glibc_has_attribute (__warn_unused_result__)
        |                                                                         ^
  In file included from /usr/include/features.h:484,
                   from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                   from /usr/include/stdio.h:27,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/global.h:9,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/include/config.h:594,
                   from /tmp/pip-install-hwyxcz4x/nle_42fbe32f9d4b45f0aea77cd910f25dc0/src/dlb.c:5:
  /usr/include/x86_64-linux-gnu/sys/cdefs.h:345:73: error: macro "__has_attribute" requires an identifier
    345 | #if __GNUC_PREREQ (3,4) || __glibc_has_attribute (__warn_unused_result__)
        |                                                                         ^
  gmake[2]: *** [util/CMakeFiles/dlb.dir/build.make:118: util/CMakeFiles/dlb.dir/__/src/alloc.c.o] Error 1
  gmake[2]: *** Waiting for unfinished jobs....
  gmake[2]: *** [util/CMakeFiles/dlb.dir/build.make:104: util/CMakeFiles/dlb.dir/panic.c.o] Error 1
  gmake[2]: *** [util/CMakeFiles/recover.dir/build.make:76: util/CMakeFiles/recover.dir/recover.c.o] Error 1
  gmake[1]: *** [CMakeFiles/Makefile2:410: util/CMakeFiles/recover.dir/all] Error 2
  gmake[1]: *** Waiting for unfinished jobs....
  gmake[2]: *** [util/CMakeFiles/dlb.dir/build.make:76: util/CMakeFiles/dlb.dir/dlb_main.c.o] Error 1
  gmake[2]: *** [util/CMakeFiles/dlb.dir/build.make:90: util/CMakeFiles/dlb.dir/__/src/dlb.c.o] Error 1
  gmake[1]: *** [CMakeFiles/Makefile2:384: util/CMakeFiles/dlb.dir/all] Error 2
  gmake[2]: *** [util/CMakeFiles/makedefs.dir/build.make:104: util/CMakeFiles/makedefs.dir/__/src/objects.c.o] Error 1
  gmake[2]: *** Waiting for unfinished jobs....
  gmake[2]: *** [util/CMakeFiles/makedefs.dir/build.make:90: util/CMakeFiles/makedefs.dir/__/src/monst.c.o] Error 1
  [  7%] Linking C static library libfcontext.a
  gmake[2]: *** [util/CMakeFiles/makedefs.dir/build.make:76: util/CMakeFiles/makedefs.dir/makedefs.c.o] Error 1
  gmake[1]: *** [CMakeFiles/Makefile2:306: util/CMakeFiles/makedefs.dir/all] Error 2
  [  8%] Linking C static library libnethackdl.a
  [  8%] Built target fcontext
  [  8%] Built target nethackdl
  gmake: *** [Makefile:136: all] Error 2
  ----------------------------------------
  ERROR: Failed building wheel for nle
Failed to build nle
ERROR: Could not build wheels for nle, which is required to install pyproject.toml-based projects

Environment:

(nle) ➜  python --version
Python 3.8.12
(nle) ➜  gcc --version
gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(nle) ➜  cmake --version
cmake version 3.22.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).

(nle) ➜  uname -a
Linux pop-os 5.15.8-76051508-generic #202112141040~1639505278~21.10~0ede46a SMP Tue Dec 14 22:38:29 U x86_64 x86_64 x86_64 GNU/Linux
heiner commented 2 years ago

I'm not certain what's going on here but the issue seems to originate from system headers on your machine?

heiner commented 2 years ago

My suspicion is that there's incompatible headers installed on your system, e.g. both GCC and clang? Hard to say from here and Google doesn't seem to yield anything meaningful.

maximzxc commented 2 years ago

I see, hopped that you might know something... I also have no idea, looks weird indeed, it's a newly installed OS, had no chance to pollute it with garbage yet. I have only gcc now.

➜  which clang
clang not found
➜  which gcc
/usr/bin/gcc
heiner commented 2 years ago

If you manage to reproduce this inside Dockers I'd be interested and will follow up (perhaps more likely in the new year).

maximzxc commented 2 years ago

It seems like the issue is caused by gblic starting from 2.34 version, I can reproduce it in docker with ubuntu:impish-20211102 for example, but with ubuntu:hirsute-20211107 all works well. The only problem is that it's almost impossible to downgrade glibc in ubuntu locally and working with docker is not always convenient.

heiner commented 2 years ago

That's great info, thanks!

Can you share the Docker file(s)?

maximzxc commented 2 years ago

Sure, you can find them here: https://gist.github.com/maximzxc/c9666de324737b4e27d12dbe56ab6d71

hirsute uses glibc 2.33 by default impish uses glibc 2.34 by default

heiner commented 2 years ago

Hey @maximzxc,

Thanks again for the report and the reproduction example.

It turns out the same issue shows up in upstream NetHack. Or at least it did, until commit https://github.com/NetHack/NetHack/commit/1cb5dc04605daa45f1c6a647c17442ed7ff2fe4e in early January (I wonder if this is a coincidence; it probably is). At any rate, this should be fixed in #299.

To test this, note that you'd have to change your Dockerfile slightly to use the local sources:

FROM ubuntu:impish-20211102

ARG PYTHON_VERSION=3.8
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -yq \
        bison \
        build-essential \
        cmake \
        curl \
        flex \
        git \
        libbz2-dev \
        ninja-build \
        wget

WORKDIR /opt/conda_setup

RUN curl -o miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
     chmod +x miniconda.sh && \
     ./miniconda.sh -b -p /opt/conda && \
     /opt/conda/bin/conda install -y python=$PYTHON_VERSION && \
     /opt/conda/bin/conda clean -ya
ENV PATH /opt/conda/bin:$PATH

COPY . /opt/nle/

WORKDIR /opt/nle

RUN pip install .  # NOTE CHANGE HERE.

WORKDIR /workspace

CMD ["/bin/bash"]
maximzxc commented 2 years ago

Hey @heiner,

Thank you for the fix!

It builds well now, can't test anything yet because after the build I got a new error:

Successfully built 38d060ce4acc
Successfully tagged nle:latest
➜  nle git:(heiner/fix-impish-build-error) ✗ docker run -t -i nle python -c "import nle"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.8/site-packages/nle/__init__.py", line 4, in <module>
    import nle.env
  File "/opt/conda/lib/python3.8/site-packages/nle/env/__init__.py", line 5, in <module>
    from nle.env.base import NLE, DUNGEON_SHAPE
  File "/opt/conda/lib/python3.8/site-packages/nle/env/base.py", line 17, in <module>
    from nle import nethack
  File "/opt/conda/lib/python3.8/site-packages/nle/nethack/__init__.py", line 3, in <module>
    from nle._pynethack.nethack import *  # noqa: F403
ImportError: /opt/conda/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /opt/conda/lib/python3.8/site-packages/nle/_pynethack.cpython-38-x86_64-linux-gnu.so)

But probably the initial issue can be marked as fixed now. And this new one I'll try to resolve on weekends.

heiner commented 2 years ago

Hey @maximzxc,

Thanks for the quick response.

The issue you encounter here has to do with the libstdc++.so.6 in conda shadowing the system one. If you do rm /opt/conda/lib/libstdc++.so* inside the docker container things will work. (But that is perhaps not the best way to fix this.)