[GDAL] libpq.5.dylib no longer exists in MacOS 11.0

[jeroen]

Trying to load the CRAN binary packages for sf or rgdal or RPostgreSQL on MacOS 11.0 fails due to libpq.5.dylib:

 unable to load shared object '/Library/Frameworks/R.framework/Versions/4.0/Resources/library/sf/libs/sf.so':
  dlopen(/Library/Frameworks/R.framework/Versions/4.0/Resources/library/sf/libs/sf.so, 6): Library not loaded: /usr/lib/libpq.5.dylib
  Referenced from: /Library/Frameworks/R.framework/Versions/4.0/Resources/library/sf/libs/sf.so
  Reason: image not found

Even though a static libpq is available in libs-4, the CRAN binaries are still dynamically linked to the OS version on 10.13.

Perhaps the problem is that libs in libpq-9.6.9-darwin.17-x86_64.tar.gz are in an unusual subdirectory /usr/local/pgsql/lib where he linker doesn't find them, and falls back on /usr/lib/libpq.5.dylib.

I think rebuilding the R packages w/ -L/usr/local/pgsql/lib will solve the problem, or alternatively, repack libpq-9.6.9-darwin.17-x86_64.tar.gz so that headers and libs are in the usual /usr/local/{include,lib} folders.

See also: https://github.com/r-spatial/sf/issues/1502

[dankelley]

I've run into the same thing (a link above shows what I'm trying to do). Since Big Sur became available yesterday, I suspect a lot of macOS users are going to be in trouble soon.

When I do

Rscript -e 'requireNamespace("rgdal")'                                               [0/1856]
Loading required namespace: rgdal
Failed with error:  ‘unable to load shared object '/Library/Frameworks/R.framework/Versions/4.0/Resources/library/rgdal/libs/rgdal.so':
  dlopen(/Library/Frameworks/R.framework/Versions/4.0/Resources/library/rgdal/libs/rgdal.so, 6): Library not loaded: /usr/lib/libpq.5.dylib
  Referenced from: /Library/Frameworks/R.framework/Versions/4.0/Resources/library/rgdal/libs/rgdal.so
  Reason: image not found’

I see that it's looking for /usr/lib/libpq.5.dylib, which is not contained in the pre-built contents of the https://mac.r-project.org/libs-4/libpq-9.6.9-darwin.17-x86_64.tar.gz tarball, so I wonder whether I should try getting the source and doing a compile for myself.

[abuabara]

https://github.com/r-spatial/sf/issues/1536#issuecomment-727342736

[jimrpy]

The same problem...

[dankelley]

For me, the solution was to install the rgdal and sf R packages from the sources provided on CRAN. I also updated my homebrew ensemble, although I did not keep close track of which programs got updated, and in fact I don't even know if this problem was related to anything installed with brew.

[s-u]

Hot-fix applied on the build machine by symlinking pgsql/* one level down (so just re-install from CRAN), but proper fix would be to address it in the pq recipe.

[jeroen]

Thank you, I can confirm that the new binaries for sf, rgdal and RPostgreSQL now work on MacOS 11.0 Big Sur!

Yes it would be great to fix this in the recipe. Perhaps we can imitate the homebrew libpq recipe. What they do is passing --libdir and --includedir to ./configure and then also passing those again as make variables.

[gacolitti]

Reinstalling sf binary also worked for me:

install.packages("sf", type = "binary")

[s-u]

Yes, FWIW the underlying issue was that the gdal recipe only set CPPFLAGS for pgsql, but not LIBS. That was fixed by https://github.com/R-macos/recipes/commit/661f3828c44fd4fd21ad449b9bb5ec6850e5aeb7

[jeroen]

@s-u what matters most is to use -L/usr/local/pgsql/lib when building the R package.

With https://github.com/R-macos/recipes/commit/661f3828c44fd4fd21ad449b9bb5ec6850e5aeb7 the result is that the gdal executables (things under /usr/local/bin) are linked to your static pgsql. But you also need to pass this flag when building the R packages, otherwise they will still end up linking to /usr/lib/libpq.dylib.

[s-u]

GDAL records the flags and passes them down, so it will be included in rgdal, sf and friends since they ask GDAL for the flags and thus they don't need a special override. That said, the real problem is that pqsql is the whole PostgreSQL DB and we're only picking few files from here - likely we should also include libpq.pc to make sure everyone is using the correct flags and overrides are no longer necessary, but that will require some testing to make sure it works as intended.

[jeroen]

GDAL records the flags and passes them down, so it will be included in rgdal, sf and friends since they ask GDAL for the flags and thus they don't need a special override.

hmm how do they get passed down? I'm not seeing that flag in the lib/pkgconfig/gdal.pc file in gdal-3.1.4-darwin.17-x86_64.tar.gz

[s-u]

$ gdal-config --dep-libs
-lcrypto -lfreexl -liconv -L/usr/local/lib -lgeos_c -lgeos -lm -lstdc++ -lwebp -lsqlite3 -lexpat -L/usr/local/lib -lopenjp2 -L/usr/local/lib -lnetcdf -lhdf5_hl -lhdf5 -lm -lz -lcurl -lsz -lhdf5 -lmfhdf -ldf -ljpeg -lpng -lpq -llzma -lproj -lz -lpthread -ldl -L/usr/local/pgsql/lib -lsqlite3 -lsz -lpcre -lcurl -liconv -lxml2

[jeroen]

Thanks, that works (I was using pkg-config instead of gdal-config)

[s-u]

I looked into it, and the other problem is that even if we package libpq.pc it is not in the directory that pkgconfig expected so it doesn't work. So I have now changed the system to explicitly set prefix by default. This also allows custom locations, not just usr/local which we will probably use on arm to avoid conflicts with homebrew and Intel binaries.