R: Add support for R kernel (ark) on Linux

Currently ark only supports macOS and Windows. We should add Linux support for ark, and Linux CI releases, so that ark can be used with the Linux edition of Positron.

There is some work to be done for Linux, but not a tremendous amount; there is already a CI job that both builds and tests ark for Linux, and it's been passing for some time.

https://github.com/posit-dev/amalthea/blob/main/.github/workflows/amalthea-ci.yml

Ready for verification, we now ship Ark in Linux packages in our releases.

@dskard Would you mind trying out these packages and confirm that Ark is working out of the box?

Some notes from installing the deb and rpm packages:

we can use the following commands to launch a EC2 instance based test environment, using tools from the shiny-buckets repo and github cli (gh):

download linux package from main branch build https://github.com/posit-dev/positron/actions/runs/7808918016

cd /tmp/
download_dir=$(mktemp -d)
latest_run_id=$(gh run list --repo posit-dev/positron --workflow "Positron: Build Release" --status success --limit 1 --json databaseId --jq '.[].databaseId')
gh run download ${latest_run_id} --dir ${download_dir} --repo posit-dev/positron -n positron-binary-deb -n positron-binary-rpm

launch ec2 instance with vnc server

./bin/fuzzbucket-desktop -a ubuntu22-ide-prereqs-dev -v 59003

copy the binaries over to the ec2 instance

fuzzbucket-client scp ubuntu22-ide-prereqs-dev-1707934183 -r ${download_dir} __BOX__:/tmp/positron-packages

in the vnc viewer, install positron for ubuntu and debian based systems, use:

sudo apt-get install /tmp/positron-packages/positron-binary-deb/Positron-*.deb

or for redhat based systems, use:

sudo yum install /tmp/positron-packages/positron-binary-rpm/Positron-*.rpm

or for suse based systems, use:

sudo zypper install /tmp/positron-packages/positron-binary-rpm/Positron-*.rpm

launch positron
```
positron
```

Testing using the packages from build id 7998408788, https://github.com/posit-dev/positron/actions/runs/7998408788:

Positron-2024.02.0-1567.rpm
Positron-2024.02.0-1567.deb

on ubuntu22, we can install positron, but launching an R interpreter gives the following error:

R 4.2.2 starting.
Kernel exited with status 0 during startup.
/usr/share/positron/resources/app/extensions/positron-r/resources/ark/ark --connection_file /tmp/kernel-UuDAI5/connection.json --log /tmp/kernel-UuDAI5/kernel.log --startup-file /usr/share/positron/resources/app/extensions/positron-r/resources/scripts/startup.R -- --interactive --no-restore-data
2024-02-22T15:56:42.365436026Z [ark-unknown] ERROR crates/ark/src/interface.rs:200: Can't load R modules: Error evaluating base::tryCatch(import_positron(exprs = base::quote(structure(expression(
    env_bind_force <- function(env, name, value) {
        name <- as.character(name)
        local_unlock_binding(env, name)
        original <- env[[name]]
        assign(name, value, envir = env)
        invisible(original)
    }, local_unlock_binding <- function(env, name, frame = parent.frame()) {
        if (name %in% names(env) && bindingIsLocked(name, env)) {
            unlockBinding(name, env)
            defer(lockBinding(name, env), envir = frame)
        }
    }), srcfile = <environment>, wholeSrcref = structure(c(1L, 
0L, 25L, 0L, 0L, 0L, 1L, 25L), srcfile = <environment>, class = "srcref")))), 
    error = identity, interrupt = identity): unable to load shared object '/opt/R/4.2.2/lib/R/library/utils/libs/utils.so':
  libR.so: cannot open shared object file: No such file or directory

Occurred at:
   0: ark::modules::initialize
   1: ark::interface::start_r

Stack backtrace:
   0: ark::modules::initialize
   1: ark::interface::start_r
   2: ark::main
   3: std::sys_common::backtrace::__rust_begin_short_backtrace
   4: std::rt::lang_start::{{closure}}
   5: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:284:13
      std::panicking::try::do_call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
      std::panicking::try
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
      std::panic::catch_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
      std::panicking::try
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
      std::panic::catch_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
      std::rt::lang_start_internal
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148:20
   6: main
   7: <unknown>
   8: __libc_start_main
   9: _start
2024-02-22T15:56:42.374691618Z [ark-unknown] ERROR crates/ark/src/interface.rs:206: Error registering some hooks: Error evaluating base::tryCatch(.ps.register_all_hooks(), error = identity, interrupt = identity): could not find function ".ps.register_all_hooks"

On rhel9, we can't install positron due to libstdc++ version differences.

[ec2-user@ip-172-98-3-98 ~]$ sudo yum install /tmp/positron-packages/positron-binary-rpm/Positron-*.rpm
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:02:26 ago on Thu 22 Feb 2024 03:46:34 PM UTC.
Error:
 Problem: conflicting requests
  - nothing provides libstdc++.so.6(GLIBCXX_3.4.30)(64bit) needed by positron-1.86.0-1708569804.el8.x86_64 from @commandline
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[ec2-user@ip-172-98-3-98 ~]$

on rhel8, I think the RPM file is listing too many version of libc.so and libstdc++.so as dependencies.

[ec2-user@ip-172-98-2-223 ~]$ sudo yum install /tmp/positron-packages/positron-binary-rpm/Positron-*.rpm
Extra Packages for Enterprise Linux 8 - x86_64                                      79 kB/s |  26 kB     00:00
Extra Packages for Enterprise Linux 8 - x86_64                                     7.5 MB/s |  16 MB     00:02
google-chrome                                                                                             13 kB/s | 1.3 kB     00:00
google-chrome                                                                                             16 kB/s | 3.6 kB     00:00
Red Hat Enterprise Linux 8 for x86_64 - AppStream from RHUI (RPMs)                                        35 kB/s | 4.5 kB     00:00
Red Hat Enterprise Linux 8 for x86_64 - AppStream from RHUI (RPMs)                                        62 MB/s |  59 MB     00:00
Red Hat Enterprise Linux 8 for x86_64 - BaseOS from RHUI (RPMs)                                           85 kB/s | 4.1 kB     00:00
Red Hat Enterprise Linux 8 for x86_64 - BaseOS from RHUI (RPMs)                                           78 MB/s |  66 MB     00:00
Red Hat Ansible Engine 2 for RHEL 8 (RPMs) from RHUI                                                      97 kB/s | 4.0 kB     00:00
RHUI Client Configuration Server 8                                                                        20 kB/s | 1.5 kB     00:00
Error:
 Problem: conflicting requests
  - nothing provides libc.so.6(GLIBC_2.32)(64bit) needed by positron-1.86.0-1708569804.el8.x86_64 from @commandline
  - nothing provides libc.so.6(GLIBC_2.33)(64bit) needed by positron-1.86.0-1708569804.el8.x86_64 from @commandline
  - nothing provides libc.so.6(GLIBC_2.34)(64bit) needed by positron-1.86.0-1708569804.el8.x86_64 from @commandline
  - nothing provides libm.so.6(GLIBC_2.29)(64bit) needed by positron-1.86.0-1708569804.el8.x86_64 from @commandline
  - nothing provides libstdc++.so.6(GLIBCXX_3.4.26)(64bit) needed by positron-1.86.0-1708569804.el8.x86_64 from @commandline
  - nothing provides libstdc++.so.6(GLIBCXX_3.4.29)(64bit) needed by positron-1.86.0-1708569804.el8.x86_64 from @commandline
  - nothing provides libstdc++.so.6(GLIBCXX_3.4.30)(64bit) needed by positron-1.86.0-1708569804.el8.x86_64 from @commandline
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[ec2-user@ip-172-98-2-223 ~]$

on sles15.5, probably similar problems as rhel:

ec2-user@ip-172-98-27-161:~> sudo zypper install /tmp/positron-packages/positron-binary-rpm/Positron-*.rpm
Refreshing service 'Basesystem_Module_x86_64'.
Refreshing service 'Containers_Module_x86_64'.
Refreshing service 'Desktop_Applications_Module_x86_64'.
Refreshing service 'Development_Tools_Module_x86_64'.
Refreshing service 'Public_Cloud_Module_x86_64'.
Refreshing service 'Python_3_Module_x86_64'.
Refreshing service 'SUSE_Linux_Enterprise_Server_x86_64'.
Refreshing service 'Server_Applications_Module_x86_64'.
Refreshing service 'Web_and_Scripting_Module_x86_64'.
Retrieving repository 'SLE-Module-Basesystem15-SP5-Updates' metadata ..............................................................[done]
Building repository 'SLE-Module-Basesystem15-SP5-Updates' cache ...................................................................[done]
Retrieving repository 'SLE-Module-Containers15-SP5-Updates' metadata ..............................................................[done]
Building repository 'SLE-Module-Containers15-SP5-Updates' cache ...................................................................[done]
Retrieving repository 'SLE-Module-Public-Cloud15-SP5-Updates' metadata ............................................................[done]
Building repository 'SLE-Module-Public-Cloud15-SP5-Updates' cache .................................................................[done]
Loading repository data...
Reading installed packages...
Resolving package dependencies...

Problem: nothing provides 'libc.so.6(GLIBC_2.32)(64bit)' needed by the to be installed positron-1.86.0-1708569804.el8.x86_64
 Solution 1: do not install positron-1.86.0-1708569804.el8.x86_64
 Solution 2: break positron-1.86.0-1708569804.el8.x86_64 by ignoring some of its dependencies

Choose from above solutions by number or cancel [1/2/c/d/?] (c):
ec2-user@ip-172-98-27-161:~>

I figured out what's going on with the Ubuntu failure. First, two misconceptions that we had about https://github.com/posit-dev/amalthea/pull/205:

Loading libR.so with RTLD_GLOBAL exposes its symbols to subsequently loaded package libraries, but not its identity. This means that when a library like utils.so is loaded later on, and this library depends on libR.so, the dynamic linker doesn't know we have opened libR.so already and will attempt to find that library in the standard places.

Since we provide the symbols already it literally doesn't matter what libR.so is found at that point. It could be empty or contain the ScummVM engine instead of R. But if the linker can't find one, it will cause a loading error.

On macOS, the dependencies in package binaries are encoded with full paths that resolve into R's Framework folder, so this issue doesn't arise. On Linux, if you have some version of R installed in a standard location, then the linker will find a libR and be able to load the package libraries. If there is no installation of R, you get the error reported by Derrick:
```
unable to load shared object '/opt/R/4.2.2/lib/R/library/utils/libs/utils.so':
libR.so: cannot open shared object file: No such file or directory
```
Setting LD_LIBRARY_PATH at runtime has no effect on the dynamic linker. It needs to be set before the program is launched.

Unfortunately I couldn't find a way of encoding in the executable that we do provide libR.so to inform the linker that it shouldn't search for it.

Possible fixes I can think of:

Build and bundle an empty libR.so with ark. This means we renounce the self-contained binary and need some build-time configuration of resource location for packagers.
Go back to wrapping Ark with a shell script that sets the environment relative to R_HOME.
Set LD_LIBRARY_PATH to $R_HOME/lib before launching ark. We could adjust --install to do the same in the kernel.json file. It just needs to point to some version of R. It might become stale when the user upgrades R though, and this will cause confusion for sure among non-positron users of ark.

I like the last idea the best as it's simpler.

I'm kind of blown away that it can't find libR.so, since we supply a full absolute path to it in dlopen() https://stackoverflow.com/questions/75702960/how-to-dlopen-the-dependencies-of-a-library-that-is-to-be-dlopen-ed

For the 3rd idea, we could have some ark code that checks LD_LIBRARY_PATH at startup time to see if it is set on linux. It could do two things:

Confirm that it was set, and error if not
Check it against R_HOME (or R HOME, if that's not set), and ensure they are aligned, and error if not

Even if the 2nd isn't fully required, that would reduce some confusion when used outside Positron.

Does this mean we should actually remove our runtime setting of LD_LIBRARY_PATH? What about DYLD_FALLBACK_LIBRARY_PATH?

I'm kind of blown away that it can't find libR.so, since we supply a full absolute path to it in dlopen()

Note that this step completes successfully. We're able to open libR and expose its symbols to the linker for all subsequently loaded libraries. What we don't expose is the fact that we have opened it, so the linker will try to open it again if a library depends on it. On Unix the resolution of symbols and of dependencies is completely independent and RTLD_GLOBAL only affects the former, not the latter.

Does this mean we should actually remove our runtime setting of LD_LIBRARY_PATH? What about DYLD_FALLBACK_LIBRARY_PATH?

I think we should move both of these out of the ark binary. Actually we should move the entire sourcing of ldpaths out so that both R and java configuration properly take effect in ark. Maybe distributing an ark shell script is better after all, this way the environment is the same within positron and outside.

Regarding macOS they use a BSD toolchain or possibly have their own loader, but this area is governed by POSIX so I wouldn't expect much variation. Although it doesn't seem to matter because the CRAN binaries have hard-coded paths in their dependencies, we might want to be resilient against weird builds, and we need to make sure javaconf is properly propagated anyway. We can make a small experiment to verify that it works the same way as in Linux. I'll do it in python to better control the link/load environment.

First change the dependency of a package library from absolute to relative:

install_name_tool -change /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libR.dylib libR.dylib rlang.so

Then start python and load R with dlopen() and RTLD_GLOBAL:

from ctypes import *
CDLL("/Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libR.dylib", mode=RTLD_GLOBAL)

dlopening the package with a relative path dependency on libR then causes an error, the same as the one observed for Linux here:

CDLL("/Users/lionel/R/Library/4.2-aarch64/rlang/libs/rlang.so")
#> OSError: dlopen(/Users/lionel/R/Library/4.2-aarch64/rlang/libs/rlang.so, 0x0006): Library not loaded: libR.dylibdylib

Opening another package that still has a dependency on libR with an absolute path works fine:

CDLL("/Users/lionel/R/Library/4.2-aarch64/vctrs/libs/vctrs.so")
#> <CDLL '/Users/lionel/R/Library/4.2-aarch64/vctrs/libs/vctrs.so', handle 821c7830 at 0x105492ce0>

Setting the fallback path at runtime has no effect:

import os
os.environ["DYLD_FALLBACK_LIBRARY_PATH"]="/Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/"

CDLL("/Users/lionel/R/Library/4.2-aarch64/rlang/libs/rlang.so")
#> OSError: dlopen(/Users/lionel/R/Library/4.2-aarch64/rlang/libs/rlang.so, 0x0006): Library not loaded: libR.dylib

By the way I think the reason I did not observe this failure on Nathalie's laptop is that I had installed R-devel in /usr/local (but I haven't confirmed this).

Moving to 2024.10 since we need this for Workbench integration (Workbench often runs on RHEL or other RedHat derived distributions)

https://dailies.rstudio.com/ lists all of the Linux distributions supported on Posit Workbench.

We can now close this issue:

The linking to libR issue was fixed in https://github.com/posit-dev/positron/pull/2809 and will further be improved as part of https://github.com/posit-dev/positron/issues/4048.
Support for Workbench platforms is discussed in https://github.com/posit-dev/positron/issues/4506 and https://github.com/posit-dev/positron/issues/4507.

posit-dev / positron

R: Add support for R kernel (ark) on Linux #1619