Error when importing fiona on an M1 Mac.

lucasbrightnight commented 4 months ago

Expected behavior and actual behavior.

When loading fiona in my venv using either terminal or in an IDE, I hit an error. Others on my team encounter the same error. It appears as if it is trying to load libpcre2-8.0.dylib from /usr/lib however on an M1 Mac, homebrew installs this and other related files in /opt/homebrew/lib.

I poetry installed fiona v1.9.6 and confirmed that all libpcre2 related files (i.e. PCRE2 library) have been brew installed correctly in /opt/homebrew/lib.

Steps to reproduce the problem.

>>> import fiona

Python 3.10.0 (v3.10.0:b494f5935c, Oct  4 2021, 14:59:19) [Clang 12.0.5 (clang-1205.0.22.11)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.22.2 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 8.22.2
Traceback (most recent call last):
  File "/Users/...venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-4c762272c079>", line 1, in <module>
    import fiona
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/Users/.../venv/lib/python3.10/site-packages/fiona/__init__.py", line 41, in <module>
    from fiona._env import (
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
ImportError: dlopen(/Users/.../venv/lib/python3.10/site-packages/fiona/_env.cpython-310-darwin.so, 0x0002): Library not loaded: '/usr/lib/libpcre2-8.0.dylib'
  Referenced from: '/Users/.../venv/lib/python3.10/site-packages/fiona/.dylibs/libgdal.32.3.6.4.dylib'
  Reason: tried: '/usr/lib/libpcre2-8.0.dylib' (no such file)

Operating system & Hardware

Mac OS X 12.5, Apple M1 Max, 64 GB, Arm 64

Fiona and GDAL version and provenance

Fiona installed using poetry version 1.4.2. I'm unsure which exact file but it was likely one of these .whl files.

[[package]]
name = "fiona"
version = "1.9.6"
description = "Fiona reads and writes spatial data files"
category = "main"
optional = false
python-versions = ">=3.7"
files = [
    {file = "fiona-1.9.6-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:918bd27d8625416672e834593970f96dff63215108f81efb876fe5c0bc58a3b4"},
    {file = "fiona-1.9.6-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f4230eccbd896a79d1ebfa551d84bf90f512f7bcbe1ca61e3f82231321f1a532"},
    {file = "fiona-1.9.6-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:42d8a0e5570948d3821c493b6141866d9a4d7a64edad2be4ecbb89f81904baac"},
    {file = "fiona-1.9.6-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:39819fb8f5ec6d9971cb01b912b4431615a3d3f50c83798565d8ce41917930db"},
    {file = "fiona-1.9.6-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:50037c3b7a5f6f434b562b5b1a5b664f1caa7a4383b00af23cdb59bfc6ba852c"},
    {file = "fiona-1.9.6-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:c9b92aa1badb2773e7cac19bef3064d73e9d80c67c42f0928db2520a04be6f2f"},
    {file = "fiona-1.9.6.tar.gz", hash = "sha256:791b3494f8b218c06ea56f892bd6ba893dfa23525347761d066fb7738acda3b1"},
]

GDAL 3.9.0 installed via Homebrew gdal--3.9.0_1.arm64_monterey.bottle.tar.gz

sgillies commented 4 months ago

@lucasbrightnight thanks for the report! I can't reproduce this on my M3 MBP, which has libpcre2 in /opt/homebrew/lib, from Homebrew, and also in /usr/lib, which must be from macOS or XCode, yes? Can you confirm that you have or haven't installed XCode and command line tools on your computer?

lucasbrightnight commented 4 months ago

@sgillies, sure thing! As I understand it, homebrew installs to /usr/lib paths for intel chips uses whereas Apple M1 chips install to /opt/homebrew paths. I can confirm I have not installed XCode. I am using fish and bash for command line tools.

I can confirm I only have libprec2 installed in the /opt/homebrew file path and not the user/lib path.

sgillies commented 4 months ago

@lucasbrightnight my memory is that Homebrew installed to /usr/local, not /usr. On my work MBP /usr/lib/libpcre2-8.0.dylib has the same timestamp as everything in /System/Library, which suggests that it's part of macOS. I see the same thing on my personal laptop. Certainly, the macOS wheels were built on a system where /usr/lib/libpcre2-8.0.dylib existed, otherwise the library would have been vendored into the wheel like libgdal and friends. I don't have an explanation for why the file is missing from your system.

bvandepoel commented 4 months ago

Exactly the same issue here (also on an M1 mac)

geowurster commented 3 months ago

I'm experiencing the issue as well on an M1 Mac running MacOS 12 and:

$ brew --version
Homebrew 4.3.3

$ brew info pcre2 | head -1
==> pcre2: stable 10.43 (bottled), HEAD

$ xcode-select --version
xcode-select version 2395.

$ gdal-config --version
3.9.0

and a Python virtual environment with Fiona installed as a wheel:

$ python3 --version
Python 3.11.9

$ pip show setuptools pip wheel fiona | grep -E "Name|Version"

Name: setuptools
Version: 70.0.0

Name: pip
Version: 24.0

Name: wheel
Version: 0.43.0

Name: fiona
Version: 1.9.6

I have nothing matching /usr/lib/*pcre*, but Homebrew does report that the pcre2 package provides a /opt/homebrew/Cellar/pcre2/10.43/lib/libpcre2-8.0.dylib:

$ brew list pcre2
/opt/homebrew/Cellar/pcre2/10.43/bin/pcre2-config
/opt/homebrew/Cellar/pcre2/10.43/bin/pcre2grep
/opt/homebrew/Cellar/pcre2/10.43/bin/pcre2test
/opt/homebrew/Cellar/pcre2/10.43/include/ (2 files)
/opt/homebrew/Cellar/pcre2/10.43/lib/libpcre2-16.0.dylib
/opt/homebrew/Cellar/pcre2/10.43/lib/libpcre2-32.0.dylib
/opt/homebrew/Cellar/pcre2/10.43/lib/libpcre2-8.0.dylib
/opt/homebrew/Cellar/pcre2/10.43/lib/libpcre2-posix.3.dylib
/opt/homebrew/Cellar/pcre2/10.43/lib/pkgconfig/ (4 files)
/opt/homebrew/Cellar/pcre2/10.43/lib/ (8 other files)
/opt/homebrew/Cellar/pcre2/10.43/share/doc/ (109 files)
/opt/homebrew/Cellar/pcre2/10.43/share/man/ (96 files)

It is linked in /opt/homebrew/lib/libpcre2-8.0.dylib. Reinstalling with --no-binary seems to have gotten me a working version of the package.

$ pip install fiona --no-binary fiona

lucasbrightnight commented 3 months ago

pip uninstalling and then pip install fiona --no-binary fiona also seems to have worked for me

geowurster commented 3 months ago

It's worth noting that the Homebrew team only supports 3 MacOS versions, so if this problem is limited to Homebrew + MacOS 12, Homebrew will be dropping support pretty soon. The next version of MacOS was announced a few days ago, and IIRC the cycle from announcement to release is typically a 3-4 months.

sgillies commented 3 months ago

@geowurster thank you for the details. I wonder if this is related to Xcode versions and not MacOS versions. The ARM64 wheels were built on a Monterey (version 12) based instance that has Xcode 14: https://github.com/cirruslabs/macos-image-templates/pkgs/container/macos-monterey-xcode.

Remember, the way library vendoring (via delocate) works is that linked libraries not found in system locations like /usr/lib and /Sytem/Library will be copied into the wheel archive. libpcre2 was not vendored because it was found in one of the system locations on the build instance. I checked 1.10b1, 1.9.6, and 1.9.5 wheels and the library is in none of them.

Did any of you successfully use 1.9.5 wheels? Have any of you tried 1.10b1 wheels?

PCRE2 is a direct dependency of GDAL only and seems pretty likely to be used by people who use Fiona https://gdal.org/development/building_from_source.html#pcre2. This is sort of good news: it seems plausible that we could build PCRE2 from source on the build instance, link that, and it would get vendored and not mess up other dependencies.

geowurster commented 3 months ago

@sgillies I confirmed that both the 1.9.5 and 1.10b1 wheels work – at least according to a $ fio info coutwildrnp.shp --count. Loose steps:

Using the same version of Python, create a virtual environment.
Install the same version of the Python packaging tools as listed in my previous comment.
$ pip install "fiona==1.9.5"
$ fio info --count coutwildrnp.shp
$ pip install "fiona==1.10b1"
$ fio info --count coutwildrnp.shp

Two notable differences in my environment:

$ brew --version
Homebrew 4.3.7

$ brew info pcre2 | head -1
==> pcre2: stable 10.44 (bottled), HEAD

I don't see anything notable in the Homebrew pcre2 formula git history, so there must be something different in the Fiona wheel?

https://github.com/Homebrew/homebrew-core/commits/master/Formula/p/pcre2.rb

geowurster commented 3 months ago

@sgillies I should also note that I tested Fiona 1.9.6 in the same environment first to verify that the problem persisted, and then stepped through what I outlined in my previous comment.

sgillies commented 2 months ago

I believe this is sorted out in the upcoming 1.10b2 wheels.

geowurster commented 2 months ago

Yep, the 1.10b2 wheel seems to work for me.

Toblerity / Fiona