Build failing in clang on macOS

[5nefarious]

I'm trying to build habitat-sim, for which magnum-bindings appears to be a dependency. The build is failing with the following error:

FAILED: deps/magnum-bindings/src/python/corrade/CMakeFiles/corrade.dir/containers.cpp.o 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++  -Dcorrade_EXPORTS -I/Users/[REDACTED]/Code/habitat-sim/src/deps -I/Users/[REDACTED]/Code/habitat-sim/src/deps/magnum-bindings/src -I/Users/[REDACTED]/Code/habitat-sim/src/deps/magnum-bindings/src/python -Ideps/magnum-bindings/src/python -I/Users/[REDACTED]/Code/habitat-sim/src/deps/corrade/src -Ideps/corrade/src -isystem /Users/[REDACTED]/Code/habitat-sim/src/cmake/../deps/eigen-git-mirror -isystem /Users/[REDACTED]/Code/habitat-sim/src/cmake/../deps/Sophus -isystem /Users/[REDACTED]/Code/habitat-sim/src/cmake/../deps/rapidjson/include -isystem /Users/[REDACTED]/Code/habitat-sim/src/deps/pybind11/include -isystem /System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -DVERSION_INFO=\"0.1.2\" -fvisibility=hidden -O2 -g -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -fPIC -fvisibility=hidden   -Wall -Wextra -Wold-style-cast -Winit-self -Werror=return-type -Wmissing-declarations -pedantic -fvisibility=hidden -fvisibility-inlines-hidden -Wmissing-prototypes -Wno-shorten-64-to-32 -flto -std=gnu++14 -MD -MT deps/magnum-bindings/src/python/corrade/CMakeFiles/corrade.dir/containers.cpp.o -MF deps/magnum-bindings/src/python/corrade/CMakeFiles/corrade.dir/containers.cpp.o.d -o deps/magnum-bindings/src/python/corrade/CMakeFiles/corrade.dir/containers.cpp.o -c /Users/[REDACTED]/Code/habitat-sim/src/deps/magnum-bindings/src/python/corrade/containers.cpp
In file included from /Users/[REDACTED]/Code/habitat-sim/src/deps/magnum-bindings/src/python/corrade/containers.cpp:34:
/Users/[REDACTED]/Code/habitat-sim/src/deps/magnum-bindings/src/python/corrade/PyBuffer.h:55:95: error: cannot initialize an array element of type 'Py_ssize_t' (aka 'long') with an rvalue of type 'nullptr_t'
        *buffer = Py_buffer{nullptr, nullptr, 0, 0, 0, 0, nullptr, nullptr, nullptr, nullptr, nullptr};
                                                                                              ^~~~~~~

It looks like it's only using Python 2.7, which is odd. I have Python 3.7 installed, and CMake even found it:

-- Found PythonInterp: /usr/local/bin/python3 (found suitable version "3.7.2", minimum required is "3.6") 
-- Bindings being generated for python at /usr/bin/python
-- Found PythonInterp: /usr/bin/python (found version "2.7.10") 
-- pybind11 v2.3.0

[mosra]

Oh no!

I guess it's because Mac is the last system ever to have /usr/bin/python default to Python 2? :D Totally random idea: can you try changing this line to be

NAMES python3 python

instead of just python (and then wiping out habitat-sim/build)? If that works, then we'll commit that to the sim repo.

Cc: @erikwijmans

[erikwijmans]

Yeah, CMake and system python don't play nicely with each other due to system python still being 2.7 on a lot of systems (also true for ubuntu 16.04 systems). Our recommended way to build habitat-sim is in a conda/virtual/pyenv/etc environment as that'll pickup the correct version of python and will let you install without sudo.

[5nefarious]

@erikwijmans I am actually using a separate conda virtual environment for habitat. Maybe it's because Python 2.7 is installed at the system level?

@mosra I tried your suggestion and this time the build got a lot further. It failed here:

Installed /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/Pillow-6.1.0-py3.7-macosx-10.9-x86_64.egg
Searching for numpy-quaternion
Reading https://pypi.org/simple/numpy-quaternion/
Downloading https://files.pythonhosted.org/packages/a5/c4/9d54605988a118d72492001ede7b49a57b1b281cb5b9966278d0faab6730/numpy-quaternion-2019.7.23.15.26.49.tar.gz#sha256=42289bafc3292959173d93736f9c875b5c69d472934f04a76883b7b185adc42e
Best match: numpy-quaternion 2019.7.23.15.26.49
Processing numpy-quaternion-2019.7.23.15.26.49.tar.gz
Writing /var/folders/bz/250jq2210k39cdp7t9zcf77m0000gn/T/easy_install-_wkld87w/numpy-quaternion-2019.7.23.15.26.49/setup.cfg
Running numpy-quaternion-2019.7.23.15.26.49/setup.py -q bdist_egg --dist-dir /var/folders/bz/250jq2210k39cdp7t9zcf77m0000gn/T/easy_install-_wkld87w/numpy-quaternion-2019.7.23.15.26.49/egg-dist-tmp-_7fclvw6
The variable 'package_version' was not present in the environment
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/var/folders/bz/250jq2210k39cdp7t9zcf77m0000gn/T/easy_install-_wkld87w/numpy-quaternion-2019.7.23.15.26.49/setup.py", line 117, in <module>
    args, filtered_args = parser.parse_known_args(args=parseable_args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/__init__.py", line 143, in setup
    return distutils.core.setup(**attrs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/command/bdist_egg.py", line 163, in run
    self.run_command("egg_info")
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 296, in run
    self.find_sources()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
    mm.run()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 534, in run
    self.add_defaults()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/setuptools/command/egg_info.py", line 570, in add_defaults
    sdist.add_defaults(self)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/command/sdist.py", line 228, in add_defaults
    self._add_defaults_ext()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/command/sdist.py", line 311, in _add_defaults_ext
    build_ext = self.get_finalized_command('build_ext')
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/cmd.py", line 299, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/cmd.py", line 107, in ensure_finalized
    self.finalize_options()
  File "/var/folders/bz/250jq2210k39cdp7t9zcf77m0000gn/T/easy_install-_wkld87w/numpy-quaternion-2019.7.23.15.26.49/setup.py", line 109, in finalize_options
    if arg == "--":
AttributeError: 'dict' object has no attribute '__NUMPY_SETUP__'

[erikwijmans]

Do pip install numpy or pip install -r requirements.txt first

[erikwijmans]

@erikwijmans I am actually using a separate conda virtual environment for habitat. Maybe it's because Python 2.7 is installed at the system level?

Is the environment activated? i.e. if you do which python, what does it report?

[5nefarious]

@erikwijmans I did run pip install -r requirements.txt, as per the README. pip3 list gives the following packages:

Package          Version           
---------------- ------------------
attrs            19.1.0            
llvmlite         0.29.0            
numba            0.45.1            
numpy            1.17.2            
numpy-quaternion 2019.7.23.15.26.49
Pillow           6.1.0             
pip              19.1.1            
protobuf         3.9.1             
scipy            1.3.1             
setuptools       41.0.1            
six              1.12.0            
TBB              0.1               
tqdm             4.36.1            
wheel            0.33.4

which python returns /usr/bin/python. My conda environment called habitat is active.

[5nefarious]

On the other hand, which python3 gives /usr/local/bin/python3.

[erikwijmans]

Seems like something is off with your conda install. which python should return a path inside your conda environment. You should also be able to use just pip, not pip3.

[erikwijmans]

i.e.

> conda activate hsim
> which python
/Users/erikwijmans/miniconda3/envs/hsim/bin/python
> python --version
Python 3.7.4
> conda deactivate
> which python
/usr/bin/python
> python --version
Python 2.7.10

is how things behave on my system.

[mosra]

The above-proposed change was merged into Habitat as https://github.com/facebookresearch/habitat-sim/pull/246.

@5nefarious is there anything left to be done regarding this issue?

[5nefarious]

I just nuked and recreated the conda environment and tried reinstalling dependencies through conda rather than pip. Everything works now. I'm not exactly sure why, but I'm sure it has more to do with my setup than anything in these repos.

@mosra I think it's safe to mark this issue as resolved. Thanks for your help (you too, @erikwijmans).

[mosra]

Happy to hear that, thanks for letting us know! :+1: