A feature for macos

[vrurg]

On macos there is a problem with dynamic libraries. As a security measure, the os rejects to load a dynamic library unless it's located in a system directory like /usr/lib or in the same location where the executable is placed. For example, tools installed by MacPorts are placed into /opt/local/bin and they have access to /opt/local/lib.

This affects all local rakudo installations because when, say, rakudobrew installs binaries into ~/.rakudobrew/versions/moar-YYYY.MM/install/bin, all nativecalls would only be able to load what is in ...moar-YYYY.MM/install/lib. I'm personally getting used to symlink the content of MacPorts lib into rakudo's lib, but when a new port is getting installed it's easy to forget to link it.

I propose to keep this in mind and when possible implement auto-linking dylib for macos.

PS. I'm not using rakubrew yet, but plan to and would if testing will be needed.

[patrickbkr]

@vrurg I think trying to do it automatically is a bit too magical as the symlinks would not stay up to date.

Here is a possible approach I came up with, based on how I understood the problem:

We add the command rakubrew link-lib-dir <dir>. When called as rakubrew link-lib-dir /opt/local/lib it would:

• Make sure "/opt/local/lib" is not listed in the $RAKUBREW_HOME/mac-links file
• Append "/opt/local/lib" to the $RAKUBREW_HOME/mac-links file
• Delete all symlinks in <active_version>/install/lib
• For each line in $RAKUBREW_HOME/mac-links do:
  • Link all files (ignore directories and special files) in the respective folder into .../install/lib. When a file exists, ignore it.

In addition the existing rakubrew rehash and rakubrew switch commands are modified to also perform the last two steps of the above list.

Does the above make sense? Does it solve your problem? Is it a clean solution?

[vrurg]

This is similar to alternative way I was considering, except with few corrections.

• Minor: the command looks monstrous. It's better to have two commands like add-libdir, del-libdir. Still don't like it, but find for barely used ones. Otherwise it's a correct approach considering that other software could be installed locally in a different path. For example, Oracle DB client install itself in somewhat, erh, original way.
• Don't delete all symlinks. Only those pointing into the directory. Though not sure if this is necessary in the light of what rehash and switch would do. Perhaps add-link should differ from rehash in just one aspect: the registration of a new directory.
• Upon doing add-libdir existing symlinks must not be silently ignored but warned about. This would provide a user with additional information for locating possible issues if same name libs are installed in multiple location. With more experience on hands, a goof way of resolving such conflicts would be implemented. For now I would just skip this potential issue.
• BTW, rakudobrew has register feature for managing additional Rakudo installations. I think this is the way to solve the symlinking issue for such installations too.

Otherwise it should be a fantastic feature as I already was explaining to a macos user why his Rakudo fails to install OpenSSL. Also, for me it's gonna be the killer feature to finally find the bravery necessary to switch over. :)

[patrickbkr]

@vrurg I'm in the process of implementing this. Above I wrote that directories should be ignored. I'm not sure about that anymore. There are folders with libs in them in /usr/lib/ on my computer. I'd think we want to link those too. What's your opinion? If we should link, should we link the folder directly or recreate the folder structure and only link the files?

[vrurg]

We'd have to experiment on this. I'd expect symlinks to work normally and would rather try them in first place. This way you're less dependent on any possible structure changes within these dirs.

Meanwhile, I have an update. I have recently discovered that $HOME/lib is also considered safe and is picked up by the system by default. The advantage of using it is the single place where one can have everything. But this may cause unexpected breakages to those who use the directory for development.

I'm still in doubt which is the best approach: to use it or not to use.

[patrickbkr]

@vrurg Any updates on this? Is there an actual clean solution that could be implemented in rakubrew? Or should we add a paragraph to rakubrew.org detailing the problem and proposing to link stuff to $HOME/lib?

[vrurg]

Oops, just has been sorting out unread emails and found a notification about your question!

I think adding a documentation paragraph must be the best. Autolinking into rakudo directories seems like an overkill. But doing the same into $HOME/lib may break things for some users. So, let them decide how to handle this.

[patrickbkr]

@vrurg I searched a bit for some documentation on the behaviour of linking on MacOS. I found https://github.com/Homebrew/brew/issues/556#issuecomment-233621637 and https://apple.stackexchange.com/a/414625 , both seem to suggest adapting LIBRARY_PATH is the simple fix. Can you clarify? Do you have a link to some documentation about the security limitations when dynamically linking handy?

I have tried to formulate an FAQ entry to be put on rakubrew.org. Can you proof read / check for factual errors?

Raku can't access non-system libraries on MacOS

If you are on MacOS and installed a library either manually or using MacPorts, Homebrew or a similar tool, then you won't be able to link to that library from Raku without further measures.

The reason is a MacOS safety measure that prevents programs from loading libraries that are not located in (code(../lib)code), (code($HOME/lib)code) or a system directory like (code(/usr/lib)code).

To solve this one needs to link (or copy) the wanted libraries to one of those folders. Linking to (code($HOME/lib)code) is the typical choice.

So if you, for example, want to link all of (code(/opt/local/lib)code) (the default Homebrew library directory) to (code($HOME/lib)code) you can use the following command:

ln -s /opt/local/lib/* $HOME/lib/

[vrurg]

@patrickbkr The first link you've found is pretty much outdated. The second is related to compilation only.

Unfortunately, I can't currently find the links to the pages where they mention the limitation as a security measure. I'm on a mini-vacation and took a couple of minutes to look into this.

then you won't be able to link to that library

Here a anywhere else, I think it should be 'dynamically link'.

are not located in (code(../lib)code),

It's not clear, what .. is related to. The binary must be somehow mentioned.

to link (or copy)

I'd suggest using 'symlink' somehow. Not that hard linking is not possible but not desirable either. Perhaps replacing word 'Linking' in the next sentence to 'Symlinking' would do.

[vrurg]

Oh, I've rechecked it and the first link is also related to compilation.

[patrickbkr]

Have a relaxing vacation!

[patrickbkr]

@vrurg

I have searched a bit more for a source for the whitelist dlopen dir thing. Without success. The best I could find was:

https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dlopen.3.html

Looking mostly at the Paragraph "Searching". There some env vars are listed that are searched when dlopening a library. It says The environment variables are LD_LIBRARY_PATH, DYLD_LIBRARY_PATH, and DYLD_FALLBACK_LIBRARY_PATH. The first two variables have no default value. The default value of DYLD_FALL-BACK_LIBRARY_PATH is $HOME/lib;/usr/local/lib;/usr/lib. It does in no way say that using absolute paths is forbidden. To the contrary: If the main executable is a set[ug]id binary, then all environment variables are ignored, and only a full path can be used

Is it possible that the above behaviour can explain the phenomenon you observed? I.e. those environment variables aren't set adequately on your machine.

[vrurg]

I had had spend so much time on testing at the time, that was reluctant to re-test again. The man page caused a reaction in me of the kind "what? did they forget to update it?" But it seems they didn't. In fact, LD_LIBRARY_PATH does work. Again. For the clarity, here is what I was testing it with:

use NativeCall;
my $lib = $*VM.platform-library-name('png'.IO, :version(v16)); 
note "Loading ", ~$lib;
my $v = cglobal('libpng.dylib', 'png_calloc', int64);

This fails as soon as Rakudo is unable to resolve and load the library. And it did fail for me at the time, even with LD_LIBRARY_PATH in place.

My best guess would be that either macOS had a bug, or they silently pulled back their own decision after the developers outcry. Either way, I'm glad the environment variable is back again. And sorry for not properly re-testing it!

BTW, it just've crossed my mind that, if memory serves me right, the man page always had that entry in it, even when it didn't work. So, the bug version sounds more plausible now than the security version spoken out by a fellow developer.

[patrickbkr]

So there is nothing to do here right? No need to add anything to the documentation?

[vrurg]

Yes. So, I'm closing and should've done it in first place.

Thank you!

[vrurg]

@patrickbkr not everything is as good as expected. They do allow use of LD_LIBRARY_PATH. Kinda... In fact, here is what happens:

⇒ prove6 -l
t/001-meta.rakutest ....... skipped
WARNING: /Users/vrurg/src/Raku/rakudo/install/bin/rakudo is loading libcrypto in an unsafe way

The warning is gone if I symlink /opt/local/lib into $HOME/lib, literally.

I'd re-open this for a while as it's probably still worth considering.

[vrurg]

This seems to be a loooong saga... Ok, here is what I discovered today with macOS Monterey. Not sure if earlier macOS releases behave differently, but this is not much of an issue.

So, here we go. It seems that LD_LIBRARY_PATH is unreliable. Wether it works or not changes depending on phases of the Moon or whatever else they consider relevant at Apple. But the variable is not the only one supported, the consider DYLD_LIBRARY_PATH too. And it works. Except for the fact that rakubrew enforces it to a temporary location. I don't know how many libraries are affected by this, but libcrypto loading fails with WARNING: /Users/vrurg/src/Raku/rakudo-master/install/bin/rakudo is loading libcrypto in an unsafe way message.

I propose that rakubrew don't change the variable if it is set already. This would also fix cases where one would wish to have their own path set.

[patrickbkr]

@vrurg Can you clarify on what you think rakubrew does to DYLD_LIBRARY_PATH? I ask, because I am unaware of rakubrew modifying environment variables. Grepping the source for DYLD_LIBRARY_PATH yields no results.

[vrurg]

Here is the example:

🅸 ttys001 vrurg@beetle ~/s/R/App-Rakubrew ⇒ echo $DYLD_LIBRARY_PATH                          19-02-22 16:31:59 [fix-completion]
/opt/local/lib
🅸 ttys001 vrurg@beetle ~/s/R/App-Rakubrew ⇒ raku -e 'say %*ENV<DYLD_LIBRARY_PATH>'           19-02-22 16:32:02 [fix-completion]
/var/folders/y0/y7fvcmnx7477qf4wfnqgltfc0000gn/T//par-7672757267/cache-3c736d7a8c15ba9ac5e363ba7a524ca2d46d7b51:/opt/local/lib

The likely location is _perl-darwin-2level/lib/siteperl/5.26.1/PAR.pm, line 1052.

[patrickbkr]

Ouch. That seems plausible. I guess it's PAR relying on DYLD_LIBRARY_PATH somehow. In that specific example it only prefixes the env var. Is that enough to break things for you? I'll see if I can find out if there is anything I can do about PAR modifying that env var.

[patrickbkr]

To understand the problem better: Do I assume correctly, that the problem only exist when using shim mode and disappears when using env mode?

[vrurg]

Unfortunately, I'm avoiding env mode as I'm not sure it wouldn't break my setup. But it should be ok. I tested OpenSSL module. It fails when ran through a shim. But tests are passing when I do them with rakudo-m binary directly from the installation directory. The only observable difference in both cases is the temporary directory in the environment.

That's all I have for now.

[patrickbkr]

@vrurg Can you try the executable found here: https://165-234419075-gh.circle-artifacts.com/0/rakubrew-macos and check if the problem with modified DYLD_LIBRARY_PATH is fixed?

I wasn't able to test it myself, as I'm probably limited by https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html i.e. on my MacOS system DYLD_LIBRARY_PATH disappears entirely when crossing exec.

[patrickbkr]

Accidental close. It's not verified yet, that the fix works.

[patrickbkr]

ping @vrurg https://github.com/Raku/App-Rakubrew/issues/9#issuecomment-1059790664

[vrurg]

Sorry for late reply, I have little spare time and was keeping this topic unread as a reminder to myself to take care of it.

Briefly, the outcomes of the new binary are promising. I have replaced the original one and then did rakubrew rehash to update the shims. Now raku -I. t/01-basic.t, for example, works as a charm.

Worse it gets if I use prove6 or zef test. Not sure what and how resets DYLD_LIBRARY_PATH, but the test script ends up with the variable being empty.

To make sure that it is not TAP::Harnness (which both prove6 and zef are using) which plays dirty, I tested with prove6 directly like this (../rakudo is where my actual build is located): ../rakudo/install/bin/raku ../rakudo/install/share/perl6/site/bin/prove6 -I. t/01-basic.t. It works.

So, the preliminary assumption on my side is that shims still need fixing.

[vrurg]

Ah, and surely: thanks! :)

[patrickbkr]

Worse it gets if I use prove6 or zef test. Not sure what and how resets DYLD_LIBRARY_PATH, but the test script ends up with the variable being empty.

Is it possible you are observing this: https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html ?

[vrurg]

You could be right and I have pinned it down! I tested a simple script which starts itself and prints the environment variable. It does so with run routine invoking raku shim. It worked well, both invocations reported the variable value.

But then I took a closer look at prove6 and noticed that it is, apparently, using /usr/bin/env. So, I modified the script to do the secondary invocation as run "/usr/bin/env", "raku" – and voila! The second time the variable is empty.

Now it's a big problem because I see no solution to this problem. env neither has a configuration file, nor a command line option to overcome this side effect. I'm also unsure wether it's the result of the runtime protection, as in the link you provided, or wether it's something env does on its own.

[patrickbkr]

@vrurg So, what's next? Should I push that new version of Rakubrew live?

Do we need some documentation like https://docs.haskellstack.org/en/stable/faq/#why-is-dyld_library_path-ignored on our website? - They basically aknowledge the situation and recommend disabling SIP (System Integrity Protection).

Which website? This is not a Rakubrew specific problem. Should we put this on rakudo.org? I'm unsure.

[vrurg]

I think, it makes sense to push it. At least it gets one case solved.

Documenting the problem also makes full sense. I barely expect anything could be done to get it fixed about env. But OTOH, those who use rakubrew are usually do non-standard things and therefore can think out a trick to get around the problem.

Though I'm uncertain about the location to document the situation about env. I tend to think that it must be included into Rakudo distribution. But considering that few are normally checking out docs/, it's better be duplicated somewhere else. Thinking logically, the part of Raku most affected by this is NativeCall. So, perhaps corresponding section of docs.raku.org? As a platform specifics note at the end?

[patrickbkr]

The PR has been merged a month ago already. So closing.