MacOS wheel does not work on MacOS 10.15

[jhominal]

What did you try to do?

I wanted to use pycurl on my old MacBook, for which the most recent version of MacOS is 10.15

What happened?

I ran the following commands:

python3.12 -mvenv venv-test-pycurl
. venv-test-pycurl/bin/activate
pip install pycurl
python3.12 -c 'import pycurl'

Which resulted in this error:

ImportError: pycurl: libcurl link-time version (7.64.1) is older than compile-time version (8.4.0)

What did you expect to happen?

I was expecting pip install pycurl to install a working version of pycurl (or at least fail installation cleanly).

What is the PycURL version?

pip freeze tells me that I installed pycurl==7.45.3 (using pycurl-7.45.3-cp312-cp312-macosx_10_9_x86_64.whl)

What is your Python version?

Python 3.12.3

What is your operating system and its version?

I am using MacOS 10.15.7.

Is this the most recent PycURL release?

Yes

Did you check libcurl behavior?

N/A

Notes

1. I suspect that the cause of this issue is that, as cibuildwheel (in .github/workflows/ci-macos.yml) runs on MacOS 12, it simply picks up the libcurl that is present on that system, which is more recent than the version of libcurl that is present on MacOS 10.15.
2. My first attempt at installing a re-compiled version of pycurl (pip install --no-binary=pycurl pycurl) failed - an import pycurl produced the following error: ImportError: pycurl: libcurl link-time ssl backends (secure-transport, openssl) do not include compile-time ssl backend (none/other). In order to fix that, I had to set export PYCURL_SSL_LIBRARY=sectransp before running the aforementioned pip install command. (I wonder if the fact that MacOS's curl-config --ssl-backends shows a nonsensical @SSL_BACKENDS@ as a result has something to do with that compilation issue.)

In order to fix the issue, I would propose the following actions:

1. Mark the wheel produced by ci-macos.yml as being only compatible with MacOS 12 and up (as it depends on its version of libcurl being present);
2. Have some bit of logic that will automatically set sectransp as the SSL library when using pip install to retrieve and compile the library from source on MacOS, in order to facilitate the library installation experience on older MacOS installations;
3. Yank the previous binary wheel in order to prevent other people from installing it and having a non-functional install on old MacOS;

[swt2c]

Yes, I suppose the other option is that we could build our own version of curl (+ dependencies) and bundle it with the wheel on macOS. I really kind of liked the idea of using the curl shipped by Apple, though, as it made the wheels super-small and Apple would supply the updates. Using the shipped curl is painful in other ways, as you can see (e.g., the broken curl-config they shipped).

[jhominal]

I think that, as macOS < 12 do not receive security updates anymore, it would be fine for pycurl not to provide wheels for them - if the user is able to install from the source distribution with a minimum of fuss, I think that would be good enough. (That would certainly be good enough for me!)

[surya-sigtech]

@jhominal To understand better, why do you prefer sectransp as SSL_Library over openssl?

[jhominal]

@surya-sigtech If I try to set export PYCURL_SSL_LIBRARY=openssl and run pip install --no-cache --no-binary=pycurl pycurl, I get an error:

Building wheels for collected packages: pycurl
  Building wheel for pycurl (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pycurl (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Using curl-config (libcurl 7.64.1)
      Using SSL library: OpenSSL/LibreSSL/BoringSSL
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-universal2-cpython-312
      creating build/lib.macosx-10.9-universal2-cpython-312/curl
      copying python/curl/__init__.py -> build/lib.macosx-10.9-universal2-cpython-312/curl
      running build_ext
      building 'pycurl' extension
      creating build/temp.macosx-10.9-universal2-cpython-312
      creating build/temp.macosx-10.9-universal2-cpython-312/src
      clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -arch x86_64 -g -DPYCURL_VERSION=\"7.45.3\" -DHAVE_CURL_SSL=1 -DHAVE_CURL_OPENSSL=1 -DHAVE_CURL_SSL=1 -I/Users/jean/venv-test-broken-pycurl/include -I/Library/Frameworks/Python.framework/Versions/3.12/include/python3.12 -c src/docstrings.c -o build/temp.macosx-10.9-universal2-cpython-312/src/docstrings.o
      In file included from src/docstrings.c:4:
      src/pycurl.h:178:13: fatal error: 'openssl/ssl.h' file not found
      #   include <openssl/ssl.h>
                  ^~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pycurl
Failed to build pycurl
ERROR: Could not build wheels for pycurl, which is required to install pyproject.toml-based projects

In other words, I used sectransp because it actually worked when I tried it.

However, I see that when calling curl --version, I get the following output:

curl 7.64.1 (x86_64-apple-darwin19.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.39.2
Release-Date: 2019-03-27
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets

Which suggests that, with MultiSSL, libcurl was compiled with both SecureTransport and LibreSSL backends.

[surya-sigtech]

I think the error is due to missing libraries. It looks better to go with the native SecureTransport. Thanks for the info!

[jhominal]

I think the only issue is that I do not have the LibreSSL headers in my include path, and I do not know where they are.