Build GDAL binaries against libcrypto for MacOS

[kapadia]

The GDAL binary that ships with rasterio doesn't appear to be built against libcrypto or a reasonable alternative. When installing via wheels rasterio is unable to access rasters using GDAL's network filesystems (e.g. vsigs). One workaround is to build GDAL manually (or install from Homebrew) so that libcrypto is linked, then run pip install rasterio --no-binary rasterio to avoid using the prebuilt binaries.

If possible, the GDAL binary should be built against libcrypto or other appropriate library, and the wheel should be updated.

Here's an example of the limitation:

% rio info gs://....
Traceback (most recent call last):
  File "rasterio/_base.pyx", line 261, in rasterio._base.DatasetBase.__init__
  File "rasterio/_shim.pyx", line 78, in rasterio._shim.open_dataset
  File "rasterio/_err.pyx", line 216, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_NotSupportedError: CPLRSASHA256Sign() not implemented: GDAL must be built against libcrypto++ or libcrypto (openssl)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/amit/.pyenv/versions/example/bin/rio", line 8, in <module>
    sys.exit(main_group())
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/rasterio/rio/info.py", line 66, in info
    with ctx.obj['env'], rasterio.open(input) as src:
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/rasterio/env.py", line 437, in wrapper
    return f(*args, **kwds)
  File "/Users/amit/.pyenv/versions/3.7.12/envs/example/lib/python3.7/site-packages/rasterio/__init__.py", line 220, in open
    s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
  File "rasterio/_base.pyx", line 263, in rasterio._base.DatasetBase.__init__
rasterio.errors.RasterioIOError: CPLRSASHA256Sign() not implemented: GDAL must be built against libcrypto++ or libcrypto (openssl)

I just spoke with @sgillies, and volunteered to help with this issue.

[sgillies]

Here's the gist of it: libcrypto is part of OpenSSL, GDAL in the macos wheel builds uses Apple's curl and that links Apple's darwinssl, not openssl. https://github.com/rasterio/rasterio-wheels/blob/master/config.sh#L189

[sgillies]

Welp, I'm stumped. In https://github.com/rasterio/rasterio-wheels/tree/issue71 I'm using homebrew to get openssl and curl in the macos builds. That works, and libcrypto and libssl are vendored into the macos wheels, but GDAL didn't find either during the configure step and so the feature is absent from GDAL: https://github.com/rasterio/rasterio-wheels/runs/4811327137?check_suite_focus=true#step:4:756.

[vincentsarago]

FYI when trying to compile GDAL on Mac OS M1 I tried with

# openssl + crypto
sudo mkdir $PREFIX/ssl
sudo chown `whoami` $PREFIX/ssl
mkdir $BUILD_DIR/openssl \
  && curl -sfL https://github.com/openssl/openssl/archive/OpenSSL_1_1_1j.tar.gz | tar zxf - -C $BUILD_DIR/openssl --strip-components=1 \
  && cd $BUILD_DIR/openssl \
  && SDKROOT=$(xcrun --show-sdk-path --sdk macosx) MACOSX_DEPLOYMENT_TARGET=11.3 ./configure shared enable-rc5 zlib darwin64-arm64-cc --prefix=$PREFIX \
  && make -j4 --silent && make install \
  && rm -rf $BUILD_DIR/openssl

full gist over https://gist.github.com/vincentsarago/bf249e451b304314838a2214ea60e865

[sgillies]

@vincentsarago are you suggesting homebrew's openssl doesn't provide the libcrypto API that GDAL is looking for?

[vincentsarago]

@sgillies I'm not suggesting anything 😬 , when I tried to compile gdal on MAC M1 I tried to do everything outside home-brew. 🤷‍♂️

[kapadia]

I wonder if this is related to curl-config, which appears to be used when configuring GDAL.

https://github.com/rasterio/rasterio-wheels/blob/master/config.sh#L257

Despite having curl and related cryptography libraries installed with Homebrew, invoking curl-config might call out to the system curl rather than the Homebrew installed version. I can confirm on my macOS setup this is true. If I want to call curl-config against the Homebrew installed version I need something like /usr/local/opt/curl/bin/curl-config.

[sgillies]

@kapadia I believe that specifying homebrew's curl-config at https://github.com/rasterio/rasterio-wheels/blob/issue71/config.sh#L256 rules that out.

[kapadia]

@sgillies ah i see now the curl-config change was made.

Another possibility is that Homebrew's curl may not be linked to openssl. A 2019 comment on StackOverflow suggests installing via:

brew install curl-openssl

I'm giving this a try on a fork: https://github.com/kapadia/rasterio-wheels/actions/runs/1702320974

Ref: https://stackoverflow.com/a/57433103

[sgillies]

@kapadia reported to me that brew install curl-openssl didn't help. Having no mac to fool around on I am sort of at a dead end.

[kapadia]

@sgillies I'll poke around on this next week. It might be that curl is compiled as done with Linux.

[sgillies]

I think I've got a bead on this! https://github.com/rasterio/rasterio-wheels/actions/runs/1792386342 is a build with openssl and curl from source on macos and GDAL has been successfully detecting the availability of libcrypto methods.

[sgillies]

Got confirmation from @kapadia that this is resolved :tada: