Racket FFI can't find external libraries

Munksgaard commented 1 year ago

Describe the bug

When running a racket program that depends on e.g. portaudio (in a shell that contains portaudio), racket reports an FFI error.

Steps To Reproduce

$ nix-shell -p racket portaudio pkgconfig
$ raco pkg install portaudio  # say yes to all the dependencies
$ echo "#lang racket
(require rsound)" > foo.rkt
$ racket foo.rkt

Expected behavior

The program runs without error. Instead, I get this error:

$ racket foo.rkt
rsound: Note: on Linux/unix, you need to install the libportaudio library yourself. Underlying error message: ffi-lib: could not load foreign library
  path: libportaudio.so.2
  system error: libportaudio.so.2: cannot open shared object file: No such file or directory
  context...:
   /nix/store/vk9ai7qwvn59m2nmx76z22g8hv4jyzwb-racket-8.5/share/racket/collects/racket/private/more-scheme.rkt:163:2: select-handler/no-breaks
   body of "/home/munksgaard/.local/share/racket/8.5/pkgs/portaudio/portaudio/portaudio.rkt"

If I supply the correct LD_LIBRARY_PATH, everything works as expected, so I'm guessing racket needs to be aware of where Nix puts shared libraries:

Notify maintainers

@henrytill @vrthra

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.77, NixOS, 22.11 (Raccoon), 22.11.20221107.667e558`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.11.0`
 - channels(root): `"nix2105-21.05, nixos, nixos-unstable"`
 - channels(munksgaard): `""`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Originally reported in https://github.com/soegaard/sketching/issues/92

bestlem commented 1 year ago

Does https://github.com/fractalide/racket2nix help

Note I have not used either I just have seen both things browsing recently

Munksgaard commented 1 year ago

Perhaps! I'll give it a try later. I also found this blog post which seems to have useful information: https://raimonster.com/post/2020-07-05-racket-and-nix/

lillycham commented 1 year ago

This also seems to cause DrRacket to be unusable on Darwin -

ffi-lib: could not load foreign library
  path: libX11.6.dylib
  system error: dlopen(libX11.6.dylib, 0x0006): tried: '/nix/store/bp8rmv0frh0xqlkkbh8nfydln2q4j7yn-cairo-1.16.0/lib/libX11.6.dylib' (no such file), [...]
  context...:
   /nix/store/p0n37azblcw6n7mqnr3cxlfdynagpwgp-racket-8.7/share/racket/collects/ffi/unsafe.rkt:131:0: get-ffi-lib
   body of "/nix/store/p0n37azblcw6n7mqnr3cxlfdynagpwgp-racket-8.7/share/racket/pkgs/gui-lib/mred/private/wx/gtk/x11.rkt"
   body of "/nix/store/p0n37azblcw6n7mqnr3cxlfdynagpwgp-racket-8.7/share/racket/pkgs/gui-lib/mred/private/wx/platform.rkt"

I assume this is also connected with an unrelated issue though - don't think racket GUI is supposed to use X11 on macOS?

LiberalArtist commented 1 year ago

I assume this is also connected with an unrelated issue though - don't think racket GUI is supposed to use X11 on macOS?

It's being configured with --enable-xonx, which makes it try to use X11 on OSX—I don't think you'd want to do that, though:

https://github.com/NixOS/nixpkgs/blob/2abd22bcff32b285639b9696e926a257ea6dafa0/pkgs/development/interpreters/racket/default.nix#L134

(I don't use Racket via Nix, but I effectively maintain the Racket packages for Guix: I came here after seeing a question on Racket's Discourse.)

The last time I looked, I thought Nix was wrapping Racket's executables with shell scripts to set LD_LIBRARY_PATH. I think that's what this code is doing, right? https://github.com/NixOS/nixpkgs/blob/2abd22bcff32b285639b9696e926a257ea6dafa0/pkgs/development/interpreters/racket/default.nix#L109-L114

Since I see /nix/store/bp8rmv0frh0xqlkkbh8nfydln2q4j7yn-cairo-1.16.0 in the report, maybe part of the problem could be that Racket typically distributes certain shared libraries like Cairo for Mac through its own package system. I didn't see a place where you've patched around that, though maybe you have. I vaguely recall the Homebrew people having done so, at least for some libraries. We've had discussions in the Racket community upstream about whether there's something we could do to better support such use cases: it that turns out to apply to Nix, please do open an issue.

What I would suggest is that you avoid the wrapper scripts by putting the needed shared libraries in Racket's lib-search-dirs configuration entry. One way to do that would be to symlink them into /nix/store/vk9ai7qwvn59m2nmx76z22g8hv4jyzwb-racket-8.5/lib/racket. What we do in Guix is instead edit the file etc/racket/config.rktd to list the relevant directories. To be concrete, here are the contents of my /gnu/store/z0a42imbmy6563wvdavapb6c8vbi1s1z-racket-8.8/etc/racket/config.rktd—it also reflects the fact that, in Guix, we've made our racket package take our racket-minimal package as an input, rather than build Racket twice:

#hash((absolute-installation? . #t)
      (apps-dir
       .
       "/gnu/store/z0a42imbmy6563wvdavapb6c8vbi1s1z-racket-8.8/share/applications")
      (bin-dir
       .
       "/gnu/store/z0a42imbmy6563wvdavapb6c8vbi1s1z-racket-8.8/lib/racket/bogus-untethered-bin")
      (bin-search-dirs
       .
       (#f
        "/gnu/store/wwj1idg2kdji7wwiyh4naqzkd4k8vg07-racket-minimal-8.8/lib/racket/bogus-untethered-bin"
        "/gnu/store/f0j2vd8jzfqw9wkchf1aq3288d6w6b3m-racket-vm-cs-8.8/opt/racket-vm/bin"))
      (build-stamp . "")
      (catalogs
       .
       ("https://download.racket-lang.org/releases/8.8/catalog/" #f))
      (config-tethered-apps-dir
       .
       "/gnu/store/z0a42imbmy6563wvdavapb6c8vbi1s1z-racket-8.8/share/applications")
      (config-tethered-console-bin-dir
       .
       "/gnu/store/z0a42imbmy6563wvdavapb6c8vbi1s1z-racket-8.8/bin")
      (config-tethered-gui-bin-dir
       .
       "/gnu/store/z0a42imbmy6563wvdavapb6c8vbi1s1z-racket-8.8/bin")
      (doc-dir
       .
       "/gnu/store/z0a42imbmy6563wvdavapb6c8vbi1s1z-racket-8.8/share/doc/racket")
      (doc-search-dirs
       .
       (#f
        "/gnu/store/wwj1idg2kdji7wwiyh4naqzkd4k8vg07-racket-minimal-8.8/share/doc/racket"
        "/gnu/store/f0j2vd8jzfqw9wkchf1aq3288d6w6b3m-racket-vm-cs-8.8/opt/racket-vm/doc"))
      (include-dir
       .
       "/gnu/store/z0a42imbmy6563wvdavapb6c8vbi1s1z-racket-8.8/include/racket")
      (include-search-dirs
       .
       (#f
        "/gnu/store/wwj1idg2kdji7wwiyh4naqzkd4k8vg07-racket-minimal-8.8/include/racket"
        "/gnu/store/f0j2vd8jzfqw9wkchf1aq3288d6w6b3m-racket-vm-cs-8.8/opt/racket-vm/include"))
      (installation-name . "8.8-guix")
      (lib-dir
       .
       "/gnu/store/z0a42imbmy6563wvdavapb6c8vbi1s1z-racket-8.8/lib/racket")
      (lib-search-dirs
       .
       (#f
        "/gnu/store/wwj1idg2kdji7wwiyh4naqzkd4k8vg07-racket-minimal-8.8/lib/racket"
        "/gnu/store/f0j2vd8jzfqw9wkchf1aq3288d6w6b3m-racket-vm-cs-8.8/opt/racket-vm/lib"
        "/gnu/store/gg3kycn5wfjwskx3xfkk1qscjgsvaxcn-cairo-1.16.0/lib"
        "/gnu/store/fncsrwapajvfkl76zmn6z1cxqd7hlbqf-fontconfig-minimal-2.14.0/lib"
        "/gnu/store/nb40pwd37v6i1g4b1fq4l6q4h9px3asr-glib-2.72.3/lib"
        "/gnu/store/63n15fkd8lw3892zgva2gshmz0f5awip-glu-9.0.2/lib"
        "/gnu/store/ib2n2vzqpchc3bhh9i712w5sq9zapn8d-gmp-6.2.1/lib"
        "/gnu/store/2n2kprz35a19ibs5kbjsb3k4cdl69q2w-gtk+-3.24.37/lib"
        "/gnu/store/85276hii9m2x894d5j2zhnahg8viy1sj-libjpeg-turbo-2.1.4/lib"
        "/gnu/store/zkxvwia0z25409k1kmm0jqzfk9prc8fx-libpng-1.6.37/lib"
        "/gnu/store/0hvkv5kvrk7ix29pfnbkyppbdxa7ki7n-libx11-1.8.1/lib"
        "/gnu/store/sbwx4cl5sawylx973z5b01lpks53fc0k-mesa-23.0.3/lib"
        "/gnu/store/cs3hw1wnxgijjzsd61whc8ar3qy9wjd6-mpfr-4.2.0/lib"
        "/gnu/store/rr1vbf04j27z5465wsm1kdfaw3iriz2k-pango-1.50.10/lib"
        "/gnu/store/1ccf2b3jmnjbm9b6xglp32kq7nzqjy95-unixodbc-2.3.9/lib"
        "/gnu/store/1n1ps8f58mqwcxv74jddi1n412dia26n-libedit-20191231-3.1/lib"
        "/gnu/store/69wd3pd1hd3j84xr965jj2fk2qmxn0hl-openssl-3.0.8/lib"
        "/gnu/store/4jakqiibsvrkv4jdw1wyl6racrwv9bkh-sqlite-3.39.3/lib"))
      (links-file
       .
       "/gnu/store/z0a42imbmy6563wvdavapb6c8vbi1s1z-racket-8.8/lib/racket/links.rktd")
      (links-search-files
       .
       (#f
        "/gnu/store/wwj1idg2kdji7wwiyh4naqzkd4k8vg07-racket-minimal-8.8/lib/racket/links.rktd"
        "/gnu/store/f0j2vd8jzfqw9wkchf1aq3288d6w6b3m-racket-vm-cs-8.8/opt/racket-vm/share/links.rktd"))
      (man-dir
       .
       "/gnu/store/z0a42imbmy6563wvdavapb6c8vbi1s1z-racket-8.8/share/man")
      (man-search-dirs
       .
       (#f
        "/gnu/store/wwj1idg2kdji7wwiyh4naqzkd4k8vg07-racket-minimal-8.8/share/man"
        "/gnu/store/f0j2vd8jzfqw9wkchf1aq3288d6w6b3m-racket-vm-cs-8.8/opt/racket-vm/share/man"))
      (pkgs-dir
       .
       "/gnu/store/z0a42imbmy6563wvdavapb6c8vbi1s1z-racket-8.8/lib/racket/pkgs")
      (pkgs-search-dirs
       .
       (#f
        "/gnu/store/wwj1idg2kdji7wwiyh4naqzkd4k8vg07-racket-minimal-8.8/lib/racket/pkgs"
        "/gnu/store/f0j2vd8jzfqw9wkchf1aq3288d6w6b3m-racket-vm-cs-8.8/opt/racket-vm/share/pkgs"))
      (share-dir
       .
       "/gnu/store/z0a42imbmy6563wvdavapb6c8vbi1s1z-racket-8.8/share/racket")
      (share-search-dirs
       .
       (#f
        "/gnu/store/wwj1idg2kdji7wwiyh4naqzkd4k8vg07-racket-minimal-8.8/share/racket"
        "/gnu/store/f0j2vd8jzfqw9wkchf1aq3288d6w6b3m-racket-vm-cs-8.8/opt/racket-vm/share")))

I've been slowly working on how to make Racket packages usable as Guix packages: these changes to the Guix packaging were early results of that effort. I've always intended to try to contribute something similar to Nix one it works robustly, but I'd be happy to talk before then if it would help.

P.S.

I think https://github.com/NixOS/nixpkgs/issues/230299 is a duplicate of this issue.

Also, I hope this patching: https://github.com/NixOS/nixpkgs/blob/2abd22bcff32b285639b9696e926a257ea6dafa0/pkgs/development/interpreters/racket/default.nix#L92-L99

Should be unnecessary since https://github.com/racket/racket/issues/3707 was fixed upstream: we no longer have to do that in Guix, at least. Ideally your force-cs-variant.patch would also have been fixed by https://github.com/racket/racket/commit/1629887071fe3cc8fe8af0a7aa0d3912509cb058, though it's possible the wrapper script could create further complications.

lunik1 commented 1 year ago

I was having issues with racket language server on macOS crashing because it was unable to find X11 libraries. Replacing --enable-xonx with --enable-prefix fixed it.

Here's the error message:

Process racket-langserver stderr finished
ffi-lib: could not load foreign library
  path: libX11.6.dylib
  system error: dlopen(libX11.6.dylib, 0x0006): tried: '/nix/store/wh9b3qchfz22ar7gjraza7fkj6nqxnw0-cairo-1.16.0/lib/libX11.6.dylib' (no such file), [snip]

NixOS / nixpkgs