search

kiyo-masui / bitshuffle

Filter for improving compression of typed binary data.
Other
219 stars 76 forks source link

Can't build bitshuffle on macOs 10.15 with clang 11.0.0 #88

Closed nritsche closed 3 years ago

nritsche commented 4 years ago

When the github macos runner bumped clang to 11.0.0, we were not able to build bitshuffle anymore in https://github.com/kotekan/kotekan/pull/870/checks?check_run_id=1334216419

The build seems to fail because of -Werr, but I can't see where bitshuffle is set to build with that option. If possible -Werr should only be used for development and CI, since it breaking the build here is only annoying, not helpful.

Run git clone https://github.com/kiyo-masui/bitshuffle.git bitshuffle
  git clone https://github.com/kiyo-masui/bitshuffle.git bitshuffle
  cd bitshuffle && git pull
  python3 setup.py install --h5plugin --h5plugin-dir=/usr/local/opt/hdf5@1.10/lib/plugin
  shell: /bin/bash -e {0}
  env:
    IMG_CORE: docker.pkg.github.com/kotekan/kotekan/kotekan-core
    IMG_IWYU: docker.pkg.github.com/kotekan/kotekan/kotekan-iwyu
    PYTEST_TIMEOUT: 60
    HDF5_DIR: /usr/local/opt/hdf5@1.10/
##[debug]/bin/bash -e /Users/runner/work/_temp/3ce6be04-01dd-4a18-a75f-3bf4f540bdd6.sh
Cloning into 'bitshuffle'...
warning: Pulling without specifying how to reconcile divergent branches is
discouraged. You can squelch this message by running one of the following
commands sometime before your next pull:

  git config pull.rebase false  # merge (the default strategy)
  git config pull.rebase true   # rebase
  git config pull.ff only       # fast-forward only

You can replace "git config" with "git config --global" to set a default
preference for all repositories. You can also pass --rebase, --no-rebase,
or --ff-only on the command line to override the configured default per
invocation.

Already up to date.
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
Can't find hdf5 with pkg-config fallback to static config
running install
running build
running build_py
creating build
creating build/lib.macosx-10.15-x86_64-3.8
creating build/lib.macosx-10.15-x86_64-3.8/bitshuffle
copying bitshuffle/__init__.py -> build/lib.macosx-10.15-x86_64-3.8/bitshuffle
creating build/lib.macosx-10.15-x86_64-3.8/bitshuffle/tests
copying bitshuffle/tests/test_h5plugin.py -> build/lib.macosx-10.15-x86_64-3.8/bitshuffle/tests
copying bitshuffle/tests/test_h5filter.py -> build/lib.macosx-10.15-x86_64-3.8/bitshuffle/tests
copying bitshuffle/tests/make_regression_tdata.py -> build/lib.macosx-10.15-x86_64-3.8/bitshuffle/tests
copying bitshuffle/tests/__init__.py -> build/lib.macosx-10.15-x86_64-3.8/bitshuffle/tests
copying bitshuffle/tests/test_ext.py -> build/lib.macosx-10.15-x86_64-3.8/bitshuffle/tests
copying bitshuffle/tests/test_regression.py -> build/lib.macosx-10.15-x86_64-3.8/bitshuffle/tests
creating build/lib.macosx-10.15-x86_64-3.8/bitshuffle/tests/data
copying bitshuffle/tests/data/regression_0.1.3.h5 -> build/lib.macosx-10.15-x86_64-3.8/bitshuffle/tests/data
running build_ext
Compiling bitshuffle/ext.pyx because it changed.
Compiling bitshuffle/h5.pyx because it changed.
[1/2] Cythonizing bitshuffle/ext.pyx
[2/2] Cythonizing bitshuffle/h5.pyx
building 'bitshuffle.ext' extension
creating build/temp.macosx-10.15-x86_64-3.8
creating build/temp.macosx-10.15-x86_64-3.8/bitshuffle
creating build/temp.macosx-10.15-x86_64-3.8/src
creating build/temp.macosx-10.15-x86_64-3.8/lz4
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -DBSHUF_VERSION_MAJOR=0 -DBSHUF_VERSION_MINOR=3 -DBSHUF_VERSION_POINT=6 -Isrc/ -Ilz4/ -I/Users/runner/work/kotekan/kotekan/bitshuffle/.eggs/numpy-1.19.3-py3.8-macosx-10.15-x86_64.egg/numpy/core/include -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/include/python3.8 -c bitshuffle/ext.c -o build/temp.macosx-10.15-x86_64-3.8/bitshuffle/ext.o
In file included from bitshuffle/ext.c:653:
In file included from /Users/runner/work/kotekan/kotekan/bitshuffle/.eggs/numpy-1.19.3-py3.8-macosx-10.15-x86_64.egg/numpy/core/include/numpy/arrayobject.h:4:
In file included from /Users/runner/work/kotekan/kotekan/bitshuffle/.eggs/numpy-1.19.3-py3.8-macosx-10.15-x86_64.egg/numpy/core/include/numpy/ndarrayobject.h:12:
In file included from /Users/runner/work/kotekan/kotekan/bitshuffle/.eggs/numpy-1.19.3-py3.8-macosx-10.15-x86_64.egg/numpy/core/include/numpy/ndarraytypes.h:1822:
/Users/runner/work/kotekan/kotekan/bitshuffle/.eggs/numpy-1.19.3-py3.8-macosx-10.15-x86_64.egg/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with "          "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it with " \
 ^
bitshuffle/ext.c:2260:16: error: implicit declaration of function 'bshuf_using_NEON' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  __pyx_t_1 = (bshuf_using_NEON() != 0);
               ^
bitshuffle/ext.c:8557:3: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
  0, /*tp_print*/
  ^
/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
    Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
    ^
/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                     ^
bitshuffle/ext.c:8663:3: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
  0, /*tp_print*/
  ^
/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
    Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
    ^
/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                     ^
bitshuffle/ext.c:14480:5: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
    0,
    ^
/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
    Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
    ^
/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                     ^
4 warnings and 1 error generated.
/Users/runner/work/kotekan/kotekan/bitshuffle/.eggs/Cython-0.29.21-py3.8-macosx-10.15-x86_64.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /Users/runner/work/kotekan/kotekan/bitshuffle/bitshuffle/ext.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
/Users/runner/work/kotekan/kotekan/bitshuffle/.eggs/Cython-0.29.21-py3.8-macosx-10.15-x86_64.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /Users/runner/work/kotekan/kotekan/bitshuffle/bitshuffle/h5.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
error: command 'clang' failed with exit status 1
Error: Process completed with exit code 1.
jrs65 commented 4 years ago

It looks to me like the bshuf_using_NEON function is missing from bitshuffle_core.h. It should presumably be alongside the similar instruction set related calls here: https://github.com/kiyo-masui/bitshuffle/blob/0af8f0ea5940e66bc28f105c9027b267be44dec4/src/bitshuffle_core.h#L67

Presumably the only reason it fails to compile on clang-11 is that the warnings have gotten better.

Czaki commented 3 years ago

this is a patch for this problem https://github.com/Czaki/bitshuffle/commit/0ec5f4faab064c9f997d9fbd05e6fbc08bf60ea9

nritsche commented 3 years ago

this is a patch for this problem Czaki@0ec5f4f

Thank you, @Czaki! Can you open a pull request with this change, please?