Test for LIBS -l flags fails on OS X 11

[dmacks]

Apple's dynamic linker now has a cache of system libraries, and the actual files no longer exist. (macOS Big Sur 11.0.1 Release Notes):

New in macOS Big Sur 11.0.1, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)

That breaks how WriteMakefile processes its LIBS parameter, where for example, "-lz" is valid and leads to correct symbol resolution of what would be in libz, but there is no libz.dylib. But EU::MM reports "Warning (mostly harmless): No library found for -lz" and does not pass the flag. As a result, an XS module does not get linked against the dependent library.

Seems like there should be a fallback (or preliminary test even before bothering to search for the file?) of just trying the linker. More expensive than file searching, but also actually tests for what is actually going to happen rather than a suddenly fragile heuristic.

See fink/fink-distributions/issues/689

[Leont]

All of this logic is rather unfortunate if you ask me, but it is what it is :-/

Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache.

That does sound possible using DynaLoader

[mitchblank]

Yeah, just tripped over this after spending hours debugging a compile failure on OS/X 11.

Easy repro: on OS/X 10.15

$ perl -e 'require ExtUtils::Liblist; use Data::Dumper; print Dumper(ExtUtils::Liblist->ext("-lcurses"))'

$VAR1 = '-lcurses';
$VAR2 = '';
$VAR3 = '-lcurses';
$VAR4 = '/usr/lib';

...but on OS/X 11:

$ perl -e 'require ExtUtils::Liblist; use Data::Dumper; print Dumper(ExtUtils::Liblist->ext("-lcurses"))'
Warning (mostly harmless): No library found for -lcurses
$VAR1 = '';
$VAR2 = '';
$VAR3 = '';
$VAR4 = '';

There is some more discussion about the basic issue with OS/X 11 libraries here https://news.ycombinator.com/item?id=23612772 but the basic upshot is what @dmacks said -- the libraries exist in some real sense, you can even dlopen() them by their supposed pathname, but they don't actually exist as files on the filesystem. Great fun.

[mitchblank]

And, yes, DynaLoader does seem capable of probing for these faux library files:

$ perl -e 'require Dynaloader; use Data::Dumper; print Dumper(DynaLoader::dl_load_file("x"))'
$VAR1 = undef;
$ perl -e 'require Dynaloader; use Data::Dumper; print Dumper(DynaLoader::dl_load_file("/usr/lib/libcurses.dylib"))'
$VAR1 = '4397946183680';
$ ls -l /usr/lib/libcurses.dylib
ls: /usr/lib/libcurses.dylib: No such file or directory

[dmacks]

Good find! Just to verify, is this with Apple's perl (/usr/bin/perl5.whatever that sees various Library/Perl modules) or a built-from-source that does not see any of Apple's possible custom patching to anything?

[mitchblank]

This is just /usr/bin/perl on both machines. I suspect there isn't any perl patching involved though since the filesystem emulation happens inside of dlopen() itself.

[mohawk2]

Is this now fixed?

[Leont]

Yes, this is fixed in 7.58

[karenetheridge]

I just ran into this today building 5.32.1 on OSX 11.3 (Big Sur). I wonder if it's worthwhile pushing forward on 5.32.2 in order to bundle this fix.

Or could this be eligible for putting in Devel::PatchPerl?