search

indygreg / python-build-standalone

Produce redistributable builds of Python
BSD 3-Clause "New" or "Revised" License
1.71k stars 107 forks source link

`-Werror=unguarded-availability-new` in sysconfig prevents building of `watchdog` #210

Open asottile-sentry opened 4 months ago

asottile-sentry commented 4 months ago

admittedly this is pointing out a real warning in watchdog -- but it makes it not installable using specifically these pythons:

wget https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.12.1+20240107-x86_64-apple-darwin-pgo+lto-full.tar.zst
tar -xf cpython-3.12.1+20240107-x86_64-apple-darwin-pgo+lto-full.tar.zst
./python/install/bin/python3 -m pip wheel watchdog --no-deps

output:

...
      running build_ext
      building '_watchdog_fsevents' extension
      creating build/temp.macosx-10.9-x86_64-cpython-312
      creating build/temp.macosx-10.9-x86_64-cpython-312/src
      clang -fno-strict-overflow -DNDEBUG -g -O3 -Wall -arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -fPIC -Werror=unguarded-availability-new -DWATCHDOG_VERSION_STRING=\"3.0.0\" -DWATCHDOG_VERSION_MAJOR=3 -DWATCHDOG_VERSION_MINOR=0 -DWATCHDOG_VERSION_BUILD=0 -I/private/tmp/y/qq/python/install/include/python3.12 -c src/watchdog_fsevents.c -o build/temp.macosx-10.9-x86_64-cpython-312/src/watchdog_fsevents.o -std=c99 -pedantic -Wall -Wextra -fPIC -Wno-nullability-completeness -Wno-nullability-extension -Wno-newline-eof -Wno-error=unused-command-line-argument
      src/watchdog_fsevents.c:191:25: error: 'kFSEventStreamEventFlagItemCloned' is only available on macOS 10.13 or newer [-Werror,-Wunguarded-availability-new]
      FLAG_PROPERTY(IsCloned, kFSEventStreamEventFlagItemCloned)
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      src/watchdog_fsevents.c:163:27: note: expanded from macro 'FLAG_PROPERTY'
              if (self->flags & flag) { \
                                ^~~~
      /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/FSEvents.framework/Headers/FSEvents.h:544:3: note: 'kFSEventStreamEventFlagItemCloned' has been marked as being introduced in macOS 10.13 here, but the deployment target is macOS 10.9.0
        kFSEventStreamEventFlagItemCloned __OSX_AVAILABLE_STARTING(__MAC_10_13, __IPHONE_11_0) = 0x00400000
        ^
      src/watchdog_fsevents.c:191:25: note: enclose 'kFSEventStreamEventFlagItemCloned' in a __builtin_available check to silence this warning
      FLAG_PROPERTY(IsCloned, kFSEventStreamEventFlagItemCloned)
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      src/watchdog_fsevents.c:163:27: note: expanded from macro 'FLAG_PROPERTY'
              if (self->flags & flag) { \
                                ^~~~
      src/watchdog_fsevents.c:596:40: error: 'kFSEventStreamCreateFlagUseExtendedData' is only available on macOS 10.13 or newer [-Werror,-Wunguarded-availability-new]
                                           | kFSEventStreamCreateFlagUseExtendedData
                                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/FSEvents.framework/Headers/FSEvents.h:294:3: note: 'kFSEventStreamCreateFlagUseExtendedData' has been marked as being introduced in macOS 10.13 here, but the deployment target is macOS 10.9.0
        kFSEventStreamCreateFlagUseExtendedData __OSX_AVAILABLE_STARTING(__MAC_10_13, __IPHONE_11_0) = 0x00000040,
        ^
      src/watchdog_fsevents.c:596:40: note: enclose 'kFSEventStreamCreateFlagUseExtendedData' in a __builtin_available check to silence this warning
                                           | kFSEventStreamCreateFlagUseExtendedData
                                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      2 errors generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for watchdog
Failed to build watchdog
ERROR: Failed to build one or more wheels

[notice] A new release of pip is available: 23.3.2 -> 24.0
[notice] To update, run: /private/tmp/y/qq/python/install/bin/python3 -m pip install --upgrade pip

a workaround is to remove that flag from _sysconfigdata__darwin_darwin.py after unpacking

indygreg commented 4 months ago

Assuming you are trying to build watchdog binaries that work on macOS <10.13, this compiler error is absolutely finding a bug in watchdog where it is failing to conditionally guard symbols not available in older macOS versions!

On one hand I consider this a feature: we're preventing bugs!

On the other, we are inserting -Werror=unguarded-availability-new into the compiler flags used to build CPython itself and CPython's build system exposes that as a CFLAG that should be used to compile extension modules. This propagation is definitely an unintended side-effect. (We added this compiler flag as a guardrail to help ensure our CPython builds don't reference symbols that aren't available in the target macOS version. This is a best practice when building portable macOS binaries. TBH I'm kind of surprised the CPython builds don't do this. I can make a case that extension module builds should be using this flag to prevent end-users from building wheels that don't conform to stated ABI compatibility.)

I think it is reasonable to strip this - and other custom - compiler flag from the sysconfig data to maintain compatibility with CPython.

charliermarsh commented 4 months ago

Is this roughly: add some additional replacements to hack_sysconfig.py (in build_cpython.sh) to strip out -Werror=unguarded-availability-new?

(Do we also need to strip from CPPFLAGS?)

indygreg commented 3 months ago

Yeah, this is a modification to hack_sysconfig.py to strip flags/words from variables.

It looks like the flag is present in the following:

I'd have to look at which variables are used by Python build tools to propagate compiler flags: those are the ones we want to modify.

I'm hesitant to modify flags that denote how the Python distribution itself was built: that would constitute lying. But if a variable is overloaded to both reflect what was used and what to use, then I suppose we must modify it.