indygreg / python-zstandard

Python bindings to the Zstandard (zstd) compression library
BSD 3-Clause "New" or "Revised" License
507 stars 88 forks source link

Build fails when "zstd" is installed via `brew` #131

Closed mitchhentges closed 3 years ago

mitchhentges commented 3 years ago

To reproduce

  1. brew install python, restart shell
  2. brew install zstd
  3. pip install --user cffi (otherwise zstandard won't reproduce the exact collision in this bug)
  4. Use pip to install a version of zstandard that is incompatible with zstd in brew, e.g.: pip3 install 'zstandard==0.14.0' --no-binary :all: --user (my project is using an old version of zstandard, and not all machines are wheel-able)
  5. The compile will fail, because it'll use some files from brew's zstd package (located in /usr/local/include) and some files from the pip package. Looking at the compile command, we can see why that's the case:
    clang \
    -Wno-unused-result \
    -Wsign-compare \
    -Wunreachable-code \
    -fno-common \
    -dynamic \
    -DNDEBUG \
    -g \
    -fwrapv \
    -O3 \
    -Wall \
    -I/usr/local/include \ # <--- /usr/local/include is included first!
    -isysroot \
    /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk \
    -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include \
    -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers \
    -Izstd/common \ # <--- # Our good friends in the local package
    -Ic-ext \              # are included all the way down here,
    -Izstd \               # so Clang considers them a lower priority.
    -Izstd/compress \      #
    -Izstd/dictBuilder \   #
    -Izstd/decompress \    #
    -I/usr/local/include \
    -I/usr/local/opt/openssl@1.1/include \
    -I/usr/local/opt/sqlite/include \
    -I/usr/local/Cellar/python@3.9/3.9.1_4/Frameworks/Python.framework/Versions/3.9/include/python3.9 \
    -c \
    zstd.c \
    -o \
    build/temp.macosx-11-x86_64-3.9/zstd.o \
    -DZSTD_MULTITHREAD \
    -DZSTDLIB_VISIBILITY= \
    -DZDICTLIB_VISIBILITY= \
    -DZSTDERRORLIB_VISIBILITY= \
    -fvisibility=hidden

Moving the local package includes above the /usr/local/include one makes this compile happy :)

Why is this the case?

When setuptools builds the package, it configures its compiler. In doing so, it adds get_config_var("CFLAGS") as the first compiler arguments, and (with brew's Python) that config var has -I /usr/local/include as part of it 🤔.

What's the solution?

Unsure yet. Hopefully we can clack our heads together and find a nice solution. The existing workaround I've got is to:

brew uninstall zstd --ignore-dependencies && <install zstandard> && brew install zstd
mitchhentges commented 3 years ago

Ticket reported to brew here

mitchhentges commented 3 years ago

Brew issue closed, so the problem is resolved here, too. Cheers :beers: