search

uber / h3-py

Python bindings for H3, a hierarchical hexagonal geospatial indexing system
https://uber.github.io/h3-py
Apache License 2.0
826 stars 131 forks source link

Build Linux wheels #110

Closed bnaul closed 4 years ago

bnaul commented 4 years ago

Related to #81 and #107 but I figure it's worth its own issue, and now that the Cython branch is merged it seems like a good time. On Linux the install from source works pretty smoothly already, but 1) scikit-build doesn't play nicely with dependency ordering (see https://github.com/scikit-build/scikit-build/issues/262) and 2) the build is just kinda slow (adding it to our Docker image has roughly doubled the total build time).

I took a quick stab at this and ran into the same clock_gettime issue mentioned in #81, and the -lrt flag did fix that particular issue and the build gets pretty far:

# Start docker:
docker run -it --rm dockcross/manylinux2010-x64:latest bash

# Inside docker:
git clone https://github.com/uber/h3-py.git && cd h3-py && git submodule update --init
export PATH=$PATH:/opt/python/cp37-cp37m/bin/
export LDFLAGS="-lrt"
pip install Cython
pip wheel .

Result:

...
  [371/386] Generating C source src/h3/_cy/to_multipoly.c
  [372/386] Generating C source src/h3/_cy/edges.c
  [373/386] Generating C source src/h3/_cy/util.c
  [374/386] Generating C source src/h3/_cy/geo.c
  [375/386] Generating C source src/h3/_cy/cells.c
  [376/386] Building C object src/h3/_cy/CMakeFiles/edges.dir/edges.c.o
  [377/386] Linking C shared module src/h3/_cy/edges.cpython-37m-x86_64-linux-gnu.so
  FAILED: src/h3/_cy/edges.cpython-37m-x86_64-linux-gnu.so
  : && /opt/rh/devtoolset-8/root/usr/bin/gcc -fPIC -O3 -DNDEBUG  -lrt  -Wl,--unresolved-symbols=ignore-all -Wl,--version-script=/tmp/pip-req-build-ttv032k3/_skbuild/linux-x86_64-3.7/cmake-build/src/h3/_cy/CMakeFiles/edges-version-script.map -shared  -o src/h3/_cy/edges.cpython-37m-x86_64-linux-gnu.so src/h3/_cy/CMakeFiles/edges.dir/edges.c.o  src/h3lib/lib/libh3.a  /usr/lib64/libm.so && :
  /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: src/h3lib/lib/libh3.a(algos.c.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
  /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: src/h3lib/lib/libh3.a(coordijk.c.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
  /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: src/h3lib/lib/libh3.a(h3Index.c.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
  /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: src/h3lib/lib/libh3.a(geoCoord.c.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
  /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: src/h3lib/lib/libh3.a(faceijk.c.o): relocation R_X86_64_32S against symbol `faceCenterGeo' can not be used when making a shared object; recompile with -fPIC
  /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: src/h3lib/lib/libh3.a(baseCells.c.o): relocation R_X86_64_32S against symbol `baseCellData' can not be used when making a shared object; recompile with -fPIC
  /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: final link failed: Nonrepresentable section on output
  collect2: error: ld returned 1 exit status
  [378/386] Building C object src/h3/_cy/CMakeFiles/to_multipoly.dir/to_multipoly.c.o
  [379/386] Building C object src/h3/_cy/CMakeFiles/util.dir/util.c.o
  [380/386] Building C object src/h3/_cy/CMakeFiles/geo.dir/geo.c.o
  [381/386] Building C object src/h3/_cy/CMakeFiles/cells.dir/cells.c.o
  ninja: build stopped: subcommand failed.
  Traceback (most recent call last):
    File "/opt/_internal/cpython-3.7.7/lib/python3.7/site-packages/skbuild/setuptools_wrap.py", line 551, in setup
      cmkr.make(make_args, env=env)
    File "/opt/_internal/cpython-3.7.7/lib/python3.7/site-packages/skbuild/cmaker.py", line 482, in make
      os.path.abspath(CMAKE_BUILD_DIR)))

It seems like -fPIC is already being passed so I'm a little confused. Is anyone more knowledgable able to get these pesky last steps to pass? 🙂

ajfriend commented 4 years ago

With https://github.com/uber/h3-py/pull/112 landed, we'll soon start to work on getting wheels built via the Github Actions CI, and then uploading the artifacts to PyPI. If you wanted to take a stab in the meantime... ;)

ajfriend commented 4 years ago

Also, with #111 using pyproject.toml, does that at least help with the first issue you were seeing with scikit-build?

bnaul commented 4 years ago

Oh yeah, that did the trick, awesome! Wheels would still be cool but I think that was the biggest pain point for sure 👍

ajfriend commented 4 years ago

Awesome. We're definitely still trying to get wheels out.

Also, since it sounds like you're using the new Cython version, please do let us know if you see any issues!

bnaul commented 4 years ago

Fixed by #115, thanks @ajfriend 🙂

ajfriend commented 4 years ago

We just released v3.6.1, which should hopefully fix these install issues. Could you please try installing with pip install h3==3.6.1 and let us know how it works?