Make Windows & Mac installation put Julia runtime libraries in path

[droodman]

I have developed a Julia plugin for the commercial statistics package Stata, which is popular in economics and other fields. the plugin is infrastructure, to allow other new Stata packages to exploit the capabilities of Julia. Its core is a shared library written in C++, which embeds Julia. I want it to be easy for novice users to install Julia, install this plugin, and go.

For that to happen, I believe, the Julia /bin directory needs to be in the user's PATH. Then when the plugin runs, its calls to Julia will find their target.

It appears to me that the standard method for installing juliaup in Linux assures that. And the methods for installing it in macOS and Windows do not. On my current Windows machine, with juliaup installed from the Windows Store, "julia.exe" sits at C:\Users\drood\AppData\Local\Microsoft\WindowsApps. That's in my path, so typing "julia" at a command prompt works fine. But compiled code written to embed Julia fails to load.

On the other hand, if Windows users install Julia the pre-juliaup way and make sure to check "Add Julia to PATH". Then the environment variables get set as necessary. Mac users, meanwhile, need to install it the pre-juliaup way and type three strange lines in the shell per the platform-specific instructions.

It therefore appears to me that if I want a novice user to have a positive experience with a product built to bring them the wonders of Julia, I should tell them to use juliaup if they're on Linux, and not otherwise.

That seems suboptimal. Can installation of juliaup on Windows and Mac make sure the Julia runtime libraries are on the path too?

[davidanthoff]

That is a really good point that needs to be resolved somehow, because it clearly isn't in the current version...

I think putting things on the PATH is tricky for this, as we now have multiple Julia versions installed at the same time, and so that kind of clashes with the idea to put anything from a specific Julia version on the PATH.

Maybe one idea would be to add another "api" command to juliaup that returns the file locations for the embedding shared libraries? We already have juliaup api getconfig1 that returns a fair bit of info, and we could similarly expose the file paths for the relevant dynamic libraries?

CCing @staticfloat and @StefanKarpinski whether they have thoughts on this.

[droodman]

Thanks for taking this seriously, and for your huge contributions in general to making Julia a practical tool.

Forgive me for dramatizing the scale of the concern in this way: Yesterday afternoon, I tweeted the launch of my Stata package to bridge to Julia, as well as a package that runs on top of it in order to replace a popular command with something 10X faster (it calls FixedEffectsModels.jl). This is exciting for a section of academic social science world. The tweet has between retweeted 68 times and will probably get >100,000 "views." So this is probably an introduction to Julia for a lot of people in some disciplines where it has hardly been used!

Right now I instruct Mac and Windows users not to use juliaup. Mac users also have to type in a few lines in a terminal window per the platform-specific instructions. I'm sad that this it their introduction to Julia.

Recognizing that I'm ignorant of a lot of the considerations here, my strong recommendation would be for juliaup to add the needed PATH entries by default for the release channel. (Possibly by editing the PATH code once and changing JULIA_DIR on each update.) I suspect that a minority of users--important power users--will mulitplex. Most users will be less savvy and will just use juliaup as the easy, recommended way to install and update their sole Julia instance. I want to make their lives easy in order to make Julia more popular.

It seems to me that this approach would also be the most natural generalization of the pre-juliaup process, the best path for adding functionality without subtracting it.

If some new juliaup subcommand were added but these defaults were not set, it might be hard for me to use that. Maybe not impossible, but it would be more fragile. I'd have to make the statistical software issue a shell command, save the output to a directory to which it has write access, then read it in as if it were a flat data file, or something.

Can I ask a question while I have your attention? I am confused about why juliaup does work for my use case, in Linux. As a result, I'm not confident it will work for all Linux users. In Debian and Ubuntu 22.04 under WSL, my sole Julia-related entry in PATH is /home/droodman/.juliaup/bin. Am I right that for my Julia-embedding plugin to work, the OS must find the likes of ~/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/lib/libjulia.so.1.9? If so, how is it finding that?

Thank you!

[staticfloat]

I think there are a few separate concerns here:

1. Should juliaup's installer automatically modify PATH on Windows and macOS?
2. What should juliaup put on the PATH?
3. Will this solve @droodman's usecase of finding libjulia?

I think (1) is fairly straightfoward; we should allow the installer to put something on the PATH. On Linux (and macOS) this is already prompted:

Juliaup will be installed into the Juliaup home directory, located at:

  /Users/sabae/.juliaup

The julia, juliaup and other commands will be added to
Juliaup's bin directory, located at:

  /Users/sabae/.juliaup/bin

This path will then be added to your PATH environment variable by
modifying the profile files located at:

  /Users/sabae/.bash_profile
  /Users/sabae/.zshrc

Even better, the user is able to customize this at installation time (the default is to add the environment variables) so I think this is all well and good. Windows is a notable absence here, and I think the right thing to do is to use something like the winreg crate to add the appropriate environment variables into the registry in HKEY_CURRENT_USER\Environment, which should persist, although I'm not sure the details around appending here.

For (2), I think it makes sense to put the juliaup bin directory on the path, so that users can type julia to get the default channel, but they can also type julia +1.9 to get a particular version or other channels that they have installed, etc... This is consistent with how Linux works right now.

For (3), this really depends on how the plugin is structured, and we need more information from @droodman as to what precisely his embedding requires. It sounds like you need to load libjulia, but how are you doing that? Do you have C code that is calling dlopen()? Are you invoking julia to ask it to print out the path to libjulia, then dlopen()'ing that? Are you relying on the system dynamic linker to load libjulia on-demand when your shared library is loaded?

[droodman]

Thank you Elliot. I think the answer to your last question is the last option, "relying on the system dynamic linker to load libjulia on-demand when your shared library is loaded."

My starting point is the simple example in the embedding documentation, and I think that is what happens in that example. The major difference is that I am compiling to a shared library, not an executable. The library is in turn loaded by the commercial software Stata.

My code is cpp/julia.ado.cpp at https://github.com/droodman/julia.ado. The compilation command line is similar to that in the embedding documentation:

g++ -shared -fPIC -DSYSTEM=OPUNIX stplugin.c julia.ado.cpp -o jl.pluginLINUX64 -I$JULIA_DIR/include/julia -L$JULIA_DIR/lib -Wl,-rpath,$JULIA_DIR/lib -ljulia

[staticfloat]

Does that compilation command run once to create a binary executable that is then redistributed to users, or is that compilation command run on users' machines?

[droodman]

Run once, for distribution. Well, run four times, in different versions--Windows, Mac/Intel, Mac/ARM, Linux.

[staticfloat]

Well, that's tough. 😅

The way the dynamic linker works is that it has certain directories it searches by default (on Linux for instance, it searches /lib, /usr/lib, /usr/local/lib, etc...), it can be coerced to search additional locations via environment variables and binaries can also embed paths within themselves to search (that's what that -Wl,-rpath is doing in your compilation command). However, for distribution to users' machines, our options are limited, one of the following must be true:

1. libjulia has to be in a predefined location (e.g. /usr/local/lib). This is not great, as it means that it's impossible to have multiple versions of Julia installed at once, so we don't do it.
2. libjulia has to be in a predictable location relative to your embedding. This is what we generally expect when writing the embedding documentation; that you will bundle a Julia installation along with the embedding application. In this scenario, the RPATH hardcoded into your binary will look at some path relative to your library, and everything works out nicely.
3. As a last resort, we can use environment variables to point the system linker to additional directories.

If your plugin is not doing (2) then we have basically no choice but to contemplate (3), but I very much disapprove of users adding things to environment variables (in this case LD_LIBRARY_PATH on Linux, DYLD_FALLBACK_LIBRARY_PATH on macOS and PATH on Windows) to influence linker search paths because the results can be very difficult to debug. In this case, the results may be mostly harmless as we work hard to ensure that the only libraries in Julia's lib directory are libjulia itself, and so it is highly unlikely to conflict with other dependencies, but it is still something of a code smell (and many sysadmins' hackles will raise) to see an installer automatically modifying these environment variables.

Instead, I would propose that you ship a copy of Julia with your Stata installation. I know that likely increases the size of the installation by a huge amount, it does solve these kinds of problems. If that's a non-starter for you, is it possible to run some code in the stata process before it attempts to load your shared library? That code could look for the user's juliaup installation and automatically load the correct libjulia first.

[droodman]

Thanks for this thorough and prompt response. It sounds like there is a fundamental distinction between advertising the locations of executables, as it were, through the PATH variable, and advertising the locations of shared libraries, because the latter can have weird, bad consequences.

But can you explain to me: which of the 3 options am I effectively choosing when I instruct users to avoid juliaup?

I don't think that a user can instruct Stata to load libjulia first. And yes distributing Julia with the package is a non-starter.

I can however make Stata do something like

! julia -E"joinpath(dirname(Sys.BINDIR), \"lib\")" > tempfile

Where the ! indicates a shell command and tempfile is a file to which Stata guarantees the user has write access. Then Stata can read in tempfile.

But then I fear the complications would be beyond me. I can't assume the user has g++ installed, so I wouldn't want to try recompiling.

Would I need instead to write a plugin that is not linked to libjulia at compile time? Upon loading it, I would pass it the libjulia path, and it would engage in dynamic runtime loading of libjulia using loading code that works across all platforms? Maybe it's not so hard?

[droodman]

It seems like what symmetry demands--and maybe it's impractical--is for juliaup to do the same masquerading trick for libjulia as it does for julia.exe. There would be something called libjulia in a channel-independent spot. Programs like mine would link to it, innocently unaware of which version of libjulia the function calls were being routed to. Would it just require a symlink to the true libjulia? Or have I just brought us back to the previous issues rather than circumventing them?

Or the C++ caller need not be innocent: it could optionally specify a channel.

[davidanthoff]

But can you explain to me: which of the 3 options am I effectively choosing when I instruct users to avoid juliaup?

I think on Windows it is effectively 3). The "regular" Julia installer adds the bin folder to the PATH, and there are lots of DLLs in there that now will be picked up during dynamic-link library loading. But I agree with @staticfloat, relying on that mechanism doesn't strike me as a good solution...

Would I need instead to write a plugin that is not linked to libjulia at compile time? Upon loading it, I would pass it the libjulia path, and it would engage in dynamic runtime loading of libjulia using loading code that works across all platforms?

I think if that could be made to work it would be by far the best option. Essentially the flow then could be something like this:

1. You do ! juliaup api getconfig1 > tempfile
2. You read tempfile and extract the location of libjulia from it
3. You use something like dlopen to load that shared library

Some more thoughts on this:

• Is there a way for Stata to capture the output of running juliaup api getconfig1 without writing to a tempfile? Most programming languages have a way to just capture the output into a string, which would be easier? Not a biggy, though.
• The output of juliaup api getconfig1 right now is in JSON. Does Stata have support for parsing that? If not, we could also consider adding a new command to juliaup, say juliaup api getlibjulia1 that just outputs one line of string with the location of libjulia, so that you wouldn't have to do any parsing at all.
• There is a broader question how the Stata add-in would want to deal with the version multiplexer. Generally speaking one can ask Juliaup "what is the default Julia version right now?", but one can also ask Juliaup "What is the Julia version that I should use if I am in folder X?" The latter will pick up overrides, the env variable for configuring channels and in the future the Julia version that is stored in a manifest. So it might actually make more sense that we add something like juliaup api getlibjulia1 PATH, i.e. that you pass a path that is the path the user is working in right now, and then Juliaup will tell you which libjulia version to use.

[staticfloat]

Would I need instead to write a plugin that is not linked to libjulia at compile time? Upon loading it, I would pass it the libjulia path, and it would engage in dynamic runtime loading of libjulia using loading code that works across all platforms? Maybe it's not so hard?

Yes, I think this would be the easiest option. I assume you're writing this extension in C/C++ (because I found this), and if that is the case, I think what David is referring to here is not that difficult; just spin up an external process, capture its output, check the error code, dlopen() the result, etc... The most annoying part is that you are not going to be able to call jl_init() directly, you'll have to use dlsym() to find the address of jl_init(), cast that to a function pointer, then call that function pointer. Here's an example from the Julia codebase itself where we use lookup_symbol (a wrapper around dlsym()) to find the location of jl_repl_entrypoint, then call that.

[droodman]

Thank you, David and Elliot.

OK, I will explore the option of something like dlopen.

Stata's ability to do shell-like things is limited. About all it can do is construct command strings and send them to a child shell. So I think I would need to write output to a temporary file as above.

I understand you are building the future with the juliaup project, and I think it's great. But I think I would need to implement my application without assuming juliaup is being used. Possibly it will try juliaup first and if that fails, just try julia. But I'll want a universal solution based on the latter, at least as a backup, like the one I suggested before.

Nevertheless, something like juliaup api getlibjulia1 PATH seems like a great idea.

Stata can't handle JSON, at least not without calling Python...which it can, if Python is installed and configured properly, which I don't want to assume. But Stata has regex support, so I think it could extract what it needs from a JSON snippet.

Yes, I'm doing this in C/C++.

@staticfloat wrote:

The most annoying part is that you are not going to be able to call jl_init() directly, you'll have to use dlsym() to find the address of jl_init(), cast that to a function pointer, then call that function pointer.

I was assuming I'll need to that with all the jl_* entry points, like jl_eval_string(), jl_string_data(). Am I wrong about that?

[staticfloat]

I was assuming I'll need to that with all the jl_* entry points, like jl_eval_string(), jl_string_data(). Am I wrong about that?

Yes, that's correct. I suggest doing something like:

struct {
    (void)(*jl_eval_string)(const char *);
    (jl_value_t *)(*jl_exception_occurred)(void);
    (jl_value_t *)(*jl_call2)(jl_function_t *f, jl_value_t *a, jl_value_t *b);
    ...
} julia_fptrs;

Then fill that out with a bunch of dlsym() calls, then in the future you can just call julia_fptrs.jl_eval_string("foo").

But I think I would need to implement my application without assuming juliaup is being used.

Agreed. I actually think you probably want to just ask Julia directly for the location of its libjulia-internal, which you can do by running:

julia -e 'using Libdl; println(dlpath("libjulia-internal"))'

So if you use popen() or similar in C to run an external program and then capture its output, the above should give you a path you can directly dlopen(). Of course you will need to appropriately handle error conditions like when julia cannot be found, etc...

With regards to juliaup integration, I think it is likely enough to provide an option where a channel name can be inserted into the run command, so you may run julia +1.9 -e 'using Libdl; ...', and since juliaup already puts its julia wrapper onto PATH as discussed above, we should be good to go with the current state of affairs.

[davidanthoff]

I understand you are building the future with the juliaup project, and I think it's great. But I think I would need to implement my application without assuming juliaup is being used. Possibly it will try juliaup first and if that fails, just try julia. But I'll want a universal solution based on the latter, at least as a backup, like the one I suggested before.

Yes, totally agreed! That is essentially what we do in the Julia VS Code extension as well. We first try to run juliaup api getconfig1, and if that works we use that, but if that fails (because the user isn't using Juliaup) we fall back to other ways of finding Julia, mostly just trying to call julia directly. I think once we make Juliaup the official installer for Julia more folks will use it, but that will take time, and at the end there will probably always be some folks that don't use it, so having a fallback mechanism seems the right thing to do.

Stata can't handle JSON, at least not without calling Python

Ha, one of my missions in life is to not take dependencies on Python, so very much with you on that ;)

[davidanthoff]

With regards to juliaup integration, I think it is likely enough to provide an option where a channel name can be inserted into the run command

I actually do think we should add the option to Juliaup for a client to query which Julia version to use for a specific folder. We need that for the VS Code extension as well, so when I add that I might as well also add a version that is useful for the Stata integration.

[droodman]

Update: I'm making good progress. It turns out I needed libjulia, not libjulia-internal. And jl_get_function() is not defined in any shared library, but rather in "julia.h" in terms of other functions, so I need to load links to them instead and create my own JL_get_function().

It is working in Windows and Linux.

...but not macOS. The problem is shown in this dialog from Stata in macOS ("." is the Stata prompt):

. !echo $PATH

. !which julia

julia not found

Evidently the child shell isn't initialized by running ".zshenv", and I can imagine good reasons for that. Since the child shell can't find julia.exe, it can't query the location of libjulia, and the strategy fails.

If I do the same in Ubuntu & Debian under WSL, the PATH still shows as empty, yet which julia works, I think because the parent's PATH also gets searched, and there I am launching Stata from the prompt of a shell that has been initialized by ".bashrc".

Indeed, back in macOS, if I launch Stata from zsh--which is not easy for regular users--then it works. So the issue is that the way Stata is launched in normal usage, from the Mac GUI, the Stata application never gets visibility into the location of the Julia executable. This is in contrast with Windows, I think fundamentally because of the installation via the Windows Store, which puts julia.exe in a universally visible place.

Is the best solution for me to tell Mac users to avoid juliaup until it is on the App Store?

[droodman]

Or could juliaup on the Mac modify PATH before launching Julia?

[staticfloat]

So the issue here is that macOS doesn't source your login scripts when it starts the GUI session, which means that us adding paths into ~/.bashrc or ~/.zshrc only help if your GUI applications are launched from a shell (as then they inherit the PATH from the executable that launched them), as you already observed. Other tools like VSCode also struggle with this problem, it's not just Stata.

I think that newer macOS machines have a /etc/paths file (and a /etc/paths.d/ directory) that we can append our juliaup directory to. @droodman can you verify that you have an /etc/paths file? The main downside to this is that you need sudo privileges to add to these variables. I have not yet found a good way to set the PATH for Finder without sudo.

An alternative (which you may want to implement anyway) is for your application to manually search a few known locations. So the flow would be something like:

1. Use julia on the $PATH, if it exists.
2. Search for default juliaup installation location (e.g. ~/.julia/juliaup/bin/julia)
3. Search for default Julia.app installation directory (e.g. /Applications/Julia 1.X.app/Resources/julia/bin/julia, you can loop X from 6 to 20 just to be safe and future-proof).

Steps 1 and 2 are probably safe to do on Linux and Windows as well, just in case they haven't set up their $PATH properly.

[droodman]

OK, I'll work on that.

Yes, my ~2015 MacBook and M2 MacBook have /etc/paths.

[droodman]

Alrighty, I think I've got it all working! I've tried it on Windows, Linux, and macOS with Intel or ARM. On all platforms, it works fine with juliaup installations. I'll see what the users say.

Thank you for all your help.

[droodman]

Whoops, I'm still having a problem and would be grateful for your help. A MWE is below. I'm attempting to get a handle for a Julia function via the equivalent of jl_get_function(). The Visual Studio debugger tells me it's crashing on the last line with Exception thrown at 0x00007FFF1FB60B62 (libjulia-internal.dll) in Project1.exe: 0xC0000005: Access violation accessing location 0x0000000000000000.

It also crashes on the last line in Linux.

#include <julia.h>

#if SYSTEM==STWIN32
#include "windows.h"
#define LIBPATH "C:\\Users\\drood\\.julia\\juliaup\\julia-1.9.4+0.x64.w64.mingw32\\bin\\libjulia.dll"
#else
#include <dlfcn.h>
#define HINSTANCE void *
#define LoadLibraryA(a) dlopen((a), RTLD_LAZY)
#define GetProcAddress dlsym
#define LIBPATH "/home/droodman/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/bin/../lib/libjulia.so.1"
#endif

int main() {
    HINSTANCE hDLL = LoadLibraryA(LIBPATH);

    // pointers to needed functions in dynamically loaded libjulia
    void (*JL_init)(void);
    jl_module_t* JL_base_module;
    jl_value_t* (*JL_get_global)(jl_module_t*, jl_sym_t*);
    jl_sym_t* (*JL_symbol)(const char*);

    JL_init = (void (*)(void))GetProcAddress(hDLL, "jl_init");
    JL_base_module = (jl_module_t*)GetProcAddress(hDLL, "jl_base_module");
    JL_get_global = (jl_value_t * (*)(jl_module_t*, jl_sym_t*))GetProcAddress(hDLL, "jl_get_global");
    JL_symbol = (jl_sym_t * (*)(const char*))GetProcAddress(hDLL, "jl_symbol");

    JL_init();

    // these two line imitate the definition of jl_get_function() in julia.h
    jl_sym_t* sym = JL_symbol("sqrt"); 
    JL_get_global(JL_base_module, sym);  // <-- crashes here
}

In Linux, I'm compiling with

JULIA_DIR=/home/droodman/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu
g++ -fPIC -DSYSTEM=OPUNIX test.cpp -o test -I$JULIA_DIR/include/julia

[droodman]

Maybe I got it.