search

jetify-com / devbox

Instant, easy, and predictable development environments
https://www.jetify.com/devbox/
Apache License 2.0
7.89k stars 189 forks source link

[Bug]: ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory #1276

Open Tubo opened 12 months ago

Tubo commented 12 months ago

Current Behavior (bug) When using Python packages via requirements.txt, this time the package is ortools.

This bug was related to #710 and #760. LD_LIBRARY_PATH is /home/tubo/Sync/Projects/radscheduler/.devbox/nix/profile/default/lib

Content of LD_LIBRARY_PATH is:

libecpg.so       libecpg_compat.so       libpgcommon.a        libpgport_shlib.a     libpython3.so  python3.11
libecpg.so.6     libecpg_compat.so.3     libpgcommon_shlib.a  libpython3.11.so      pgxs           
libecpg.so.6.14  libecpg_compat.so.3.14  libpgport.a          libpython3.11.so.1.0  pkgconfig

Error messages:

============================================================ test session starts =============================================================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
django: settings: radscheduler.settings (from env)
rootdir: /home/tubo/Sync/Projects/radscheduler/radscheduler
configfile: pytest.ini
plugins: django-4.5.2
collected 0 items / 1 error                                                                                                                  

=================================================================== ERRORS ===================================================================
_______________________________________________________ ERROR collecting core/tests.py _______________________________________________________
ImportError while importing test module '/home/tubo/Sync/Projects/radscheduler/radscheduler/core/tests.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/nix/store/xlpbkwgf4nrnr56a3r0k0r32jxlxyf59-python3-3.11.4/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
radscheduler/core/tests.py:2: in <module>
    from .core import model
radscheduler/core/core.py:1: in <module>
    from ortools.sat.python import cp_model
../../../Projects/radscheduler/.devbox/virtenv/python311Packages.pip/.venv/lib/python3.11/site-packages/ortools/sat/python/cp_model.py:56: in <module>
    from ortools.sat.python import swig_helper
E   ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory
========================================================== short test summary info ===========================================================
ERROR radscheduler/core/tests.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================== 1 error in 0.15s ==============================================================

Expected Behavior (fix) Running pytest should not have this error.

Additional context

OS: KDE Neon (Ubuntu 22.04) Devbox version: 0.5.5 to 0.5.7 devbox.json:

{
  "packages": [
    "python@latest",
    "python311Packages.pip@latest",
    "openssl@latest",
    "postgresql@latest",
    "python311Packages.numpy@latest"
  ],
  "env": {
    "DJANGO_SETTINGS_MODULE": "radscheduler.settings",
    "PGPORT": "5434"
  },
  "shell": {
    "init_hook": [
      ". $VENV_DIR/bin/activate",
      "pip install -r requirements.txt --use-pep517"
    ],
    "scripts": {
      "create_db": [
        "echo \"Creating DB\"",
        "dropdb --if-exists rs_db",
        "createdb rs_db",
        "psql rs_db -c \"CREATE USER rs_user WITH PASSWORD 'secretpassword';\"",
        "python radscheduler/manage.py makemigrations",
        "python radscheduler/manage.py migrate"
      ],
      "makemigrations": "radscheduler/manage.py makemigrations",
      "migrate": "radscheduler/manage.py migrate",
      "server": "radscheduler/manage.py runserver",
      "test": "pytest radscheduler"
    }
  }
}

requirements.txt:

django==4.2.2
psycopg2==2.9.5
sqlparse==0.4.4
asgiref==3.6.0
pytest==7.4.0
pytest-django==4.5.2
ortools==9.6.2534
Lagoja commented 11 months ago

Hey @Tubo, I'm working on recreating this. Do you have an example of the test you're trying to run with pytest that triggers this issue?

Lagoja commented 11 months ago

One thing that might work is running devbox add stdenv.cc.cc.lib, which should add the libstdc++ libraries to your dev environment.

Tubo commented 11 months ago

Hey @Tubo, I'm working on recreating this. Do you have an example of the test you're trying to run with pytest that triggers this issue?

It's dead simple:

from django.test import TestCase

def test_pytest():
    assert False
Tubo commented 11 months ago

One thing that might work is running devbox add stdenv.cc.cc.lib, which should add the libstdc++ libraries to your dev environment.

Nah didn't work unfortunately:

~/P/radscheduler ❯❯❯ devbox add stdenv.cc.cc.lib

Installing package: stdenv.cc.cc.lib.

[1/1] stdenv.cc.cc.lib
[1/1] stdenv.cc.cc.lib: Success
nix: /home/tubo/Sync/Projects/radscheduler/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/nd8dpdgpjkh3g3vvcjx9qdw7mjzz0l90-nix-2.13.3/lib/libnixstore.so)
nix: /home/tubo/Sync/Projects/radscheduler/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/nd8dpdgpjkh3g3vvcjx9qdw7mjzz0l90-nix-2.13.3/lib/libnixutil.so)
nix: /home/tubo/Sync/Projects/radscheduler/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/wy40dp0iwqs9hch6c6yqxriizzgpgm3z-aws-sdk-cpp-1.9.294/lib/libaws-cpp-sdk-transfer.so)
nix: /home/tubo/Sync/Projects/radscheduler/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/wy40dp0iwqs9hch6c6yqxriizzgpgm3z-aws-sdk-cpp-1.9.294/lib/libaws-cpp-sdk-core.so)

Error: There was an error installing nix packages
source: error running "nix profile list": exit status 1

Error: There was an internal error. Run with DEVBOX_DEBUG=1 for a detailed error message, and consider reporting it at https://github.com/jetpack-io/devbox/issues

Debug information:

[DEBUG] 2023/07/19 23:22:28 go.jetpack.io/devbox/internal/boxcli/midcobra/debug.go:72: Command stderr: 
[DEBUG] 2023/07/19 23:22:28 go.jetpack.io/devbox/internal/boxcli/midcobra/debug.go:74: 
ExecutionID:c0c228a182d34f178f2cddfc74900523
error running "nix profile list": exit status 1
go.jetpack.io/devbox/internal/nix.ProfileList
        go.jetpack.io/devbox/internal/nix/profiles.go:47
go.jetpack.io/devbox/internal/nix/nixprofile.ProfileListItems
        go.jetpack.io/devbox/internal/nix/nixprofile/profile.go:28
go.jetpack.io/devbox/internal/impl.(*Devbox).extraPackagesInProfile
        go.jetpack.io/devbox/internal/impl/packages.go:419
go.jetpack.io/devbox/internal/impl.(*Devbox).tidyProfile
        go.jetpack.io/devbox/internal/impl/packages.go:357
go.jetpack.io/devbox/internal/impl.(*Devbox).syncPackagesToProfile
        go.jetpack.io/devbox/internal/impl/packages.go:249
go.jetpack.io/devbox/internal/impl.(*Devbox).ensurePackagesAreInstalled
        go.jetpack.io/devbox/internal/impl/packages.go:207
go.jetpack.io/devbox/internal/impl.(*Devbox).Add
        go.jetpack.io/devbox/internal/impl/packages.go:97
go.jetpack.io/devbox/internal/boxcli.addCmdFunc
        go.jetpack.io/devbox/internal/boxcli/add.go:68
go.jetpack.io/devbox/internal/boxcli.addCmd.func1
        go.jetpack.io/devbox/internal/boxcli/add.go:42
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.6.1/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.6.1/command.go:1044
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.6.1/command.go:968
go.jetpack.io/devbox/internal/boxcli/midcobra.(*midcobraExecutable).Execute
        go.jetpack.io/devbox/internal/boxcli/midcobra/midcobra.go:58
go.jetpack.io/devbox/internal/boxcli.Execute
        go.jetpack.io/devbox/internal/boxcli/root.go:102
go.jetpack.io/devbox/internal/boxcli.Main
        go.jetpack.io/devbox/internal/boxcli/root.go:123
main.main
        go.jetpack.io/devbox/cmd/devbox/main.go:11
runtime.main
        runtime/proc.go:250
runtime.goexit
        runtime/asm_amd64.s:1598
[DEBUG] 2023/07/19 23:22:28 go.jetpack.io/devbox/internal/impl/dir.go:22: findProjectDir: path is 
[DEBUG] 2023/07/19 23:22:28 go.jetpack.io/devbox/internal/impl/dir.go:66: finding devbox.json in dir: /home/tubo/Projects/radscheduler
Lagoja commented 11 months ago

Gotcha that's finding the libstdc file now, but it's not the right version. Let me do a little more debugging from there

aacebedo commented 10 months ago

Got the same problem. I cannot import python modules if I do not add the stdlib. But when I do that, other packages such as libsrvg are not working anymore. Unsetting LD_LIBRARY_PATH in the init hook resolve the problem for packages but breaks the import.

savil commented 10 months ago

@aacebedo we're digging into this class of errors. Could you please share some files so we can repro it: devbox.json, devbox.lock, requirements.txt, and possibly some command to repro your specific error? While a good fix may take some time, we can try to advise you on how to get unblocked.

Feel free to pull it into a new issue to keep this one scoped to Tubo's scenario?

aacebedo commented 10 months ago

Thanks @savil for the answer.

here is my devbox.yaml

{
  "packages": [
    "python310@latest",
    "python310Packages.pip@latest",
    "drawio@21.6.1",
    "inkscape@1.1.2",
    "librsvg@2.52.5",
    "xorg.xorgserver@latest",
    "plantuml@1.2023.9",
    "imagemagick@6.9.11-60",
    "stdenv.cc.cc.lib",
    "texlive.combined.scheme-full@2022.20221227"
  ],
  "shell": {
    "init_hook": [
      ". $VENV_DIR/bin/activate",
      "pip install -q -r requirements.txt"
    ]
  },
  "nixpkgs": {
    "commit": "a16f7eb56e88c8985fcc6eb81dabd6cade4e425a"
  }
}

and the requirements file:

Sphinx==5.3.0
sphinx_rtd_theme==1.1.1
sphinxcontrib-plantuml==0.25
sphinxcontrib-svg2pdfconverter==1.2.1
sphinxcontrib-drawio==0.0.16
sphinxcontrib-bibtex==2.5.0
Pillow==9.3.0
matplotlib==3.7.2

If you try it you will see that python import matplotlib is working but not the srvg-convert binary. If you add unset LD_LIBRARY_PATH to the init hook, srvg-convert works but not the python import matplotlib with the same error as OP.

savil commented 10 months ago

Thanks @aacebedo. I'm not quite able to repro because the drawio nix package appears to be broken, so it won't install (for my intel mac). I can try devbox generate dockerfile and running it in the debian container, assuming the linux package is not broken. What OS are you on?

(I have to pause for a few hours for some personal work, but can look at it a bit later today)

aacebedo commented 10 months ago

Thanks ! I am using it on NixOS.

savil commented 10 months ago

@aacebedo in a debian-container I can get a devbox shell working.

I can import matplotlib:

(.venv) (devbox) devbox@0616836e2023:/code$ cat main.py
import matplotlib

def main():
    print("hello world")

main()
(.venv) (devbox) devbox@0616836e2023:/code$ python main.py
hello world

but running rsvg-convert seems to complain...

/nix/store/0ccn204v5kds9i08x7plf4x644l6wm87-librsvg-2.52.5/bin/rsvg-convert
/nix/store/0ccn204v5kds9i08x7plf4x644l6wm87-librsvg-2.52.5/bin/rsvg-convert: /nix/store/4s21k8k7p1mfik0b33r2spq5hq7774k1-glibc-2.33-108/lib/libc.so.6: version `GLIBC_2.35' not found (required by /code/.devbox/nix/profile/default/lib/libgcc_s.so.1)
/nix/store/0ccn204v5kds9i08x7plf4x644l6wm87-librsvg-2.52.5/bin/rsvg-convert: /nix/store/4s21k8k7p1mfik0b33r2spq5hq7774k1-glibc-2.33-108/lib/libc.so.6: version `GLIBC_2.34' not found (required by /code/.devbox/nix/profile/default/lib/libgcc_s.so.1)

Is this the error you saw?

savil commented 10 months ago

For some reason rsvg is not in PATH for me...

aacebedo commented 10 months ago

It is exactly this issue. You can also try convert that exhibit a similar issue.

aacebedo commented 10 months ago

you can try adding unset LD_LIBRARY_PATH to the init hook

savil commented 10 months ago

@aacebedo I think changing the devbox.lock to use this nixpkgs commit hash for stdenv.cc.cc.lib should unblock you:

     "stdenv.cc.cc.lib": {
      "resolved": "github:NixOS/nixpkgs/c6fd903606866634312e40cceb2caee8c0c9243f#stdenv.cc.cc.lib",
      "source": "nixpkg"
    },

We're investigating more principled fixes.

aacebedo commented 10 months ago

Hi, I tried the modification but without success. convert and rsvg-convert still does not start with libc.so.6 errors

savil commented 10 months ago

@aacebedo I was discussing a similar issue with a user on Discord (is that you as well?).

In that case, something like this helped:

# main.py
import matplotlib

def main():
    print("hello world")

main()

# requirements.txt
matplotlib==3.7.2

# devbox.json
{
  "packages": [
    "python310@latest",
    "python310Packages.pip@latest",
    "drawio@latest",
    "librsvg@latest",
    "github:nixos/nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#stdenv.cc.cc.lib"
  ],
  "shell": {
    "init_hook": [
      "export DEVBOX_DEBUG=1",
      ". $VENV_DIR/bin/activate",
      "pip install -q -r requirements.txt"
    ]
  },
  "nixpkgs": {
    "commit": "73e1976309fc789706b9f306407e9e7622a57d25"
  }
}

# devbox.lock
{
  "lockfile_version": "1",
  "packages": {
    "drawio@latest": {
      "last_modified": "2023-09-10T10:53:27Z",
      "resolved": "github:NixOS/nixpkgs/78058d810644f5ed276804ce7ea9e82d92bee293#drawio",
      "source": "devbox-search",
      "version": "21.6.8"
    },
    "github:nixos/nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#stdenv.cc.cc.lib": {},
    "librsvg@latest": {
      "last_modified": "2023-09-10T10:53:27Z",
      "resolved": "github:NixOS/nixpkgs/78058d810644f5ed276804ce7ea9e82d92bee293#librsvg",
      "source": "devbox-search",
      "version": "2.56.3"
    },
    "python310@latest": {
      "last_modified": "2023-02-24T18:08:35Z",
      "plugin_version": "0.0.1",
      "resolved": "github:NixOS/nixpkgs/806075be2bdde71895359ed18cb530c4d323e6f6#python3",
      "source": "devbox-search",
      "version": "3.10.9"
    },
    "python310Packages.pip@latest": {
      "last_modified": "2023-09-10T10:53:27Z",
      "plugin_version": "0.0.1",
      "resolved": "github:NixOS/nixpkgs/78058d810644f5ed276804ce7ea9e82d92bee293#python310Packages.pip",
      "source": "devbox-search",
      "version": "23.2.1"
    }
  }

The reasoning is:

aacebedo commented 10 months ago

@aacebedo I was discussing a similar issue with a user on Discord (is that you as well?).

In that case, something like this helped:

# main.py
import matplotlib

def main():
    print("hello world")

main()

# requirements.txt
matplotlib==3.7.2

# devbox.json
{
  "packages": [
    "python310@latest",
    "python310Packages.pip@latest",
    "drawio@latest",
    "librsvg@latest",
    "github:nixos/nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#stdenv.cc.cc.lib"
  ],
  "shell": {
    "init_hook": [
      "export DEVBOX_DEBUG=1",
      ". $VENV_DIR/bin/activate",
      "pip install -q -r requirements.txt"
    ]
  },
  "nixpkgs": {
    "commit": "73e1976309fc789706b9f306407e9e7622a57d25"
  }
}

# devbox.lock
{
  "lockfile_version": "1",
  "packages": {
    "drawio@latest": {
      "last_modified": "2023-09-10T10:53:27Z",
      "resolved": "github:NixOS/nixpkgs/78058d810644f5ed276804ce7ea9e82d92bee293#drawio",
      "source": "devbox-search",
      "version": "21.6.8"
    },
    "github:nixos/nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#stdenv.cc.cc.lib": {},
    "librsvg@latest": {
      "last_modified": "2023-09-10T10:53:27Z",
      "resolved": "github:NixOS/nixpkgs/78058d810644f5ed276804ce7ea9e82d92bee293#librsvg",
      "source": "devbox-search",
      "version": "2.56.3"
    },
    "python310@latest": {
      "last_modified": "2023-02-24T18:08:35Z",
      "plugin_version": "0.0.1",
      "resolved": "github:NixOS/nixpkgs/806075be2bdde71895359ed18cb530c4d323e6f6#python3",
      "source": "devbox-search",
      "version": "3.10.9"
    },
    "python310Packages.pip@latest": {
      "last_modified": "2023-09-10T10:53:27Z",
      "plugin_version": "0.0.1",
      "resolved": "github:NixOS/nixpkgs/78058d810644f5ed276804ce7ea9e82d92bee293#python310Packages.pip",
      "source": "devbox-search",
      "version": "23.2.1"
    }
  }

The reasoning is:

  • The error seen was: Original error was: /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libc.so.6: version 'GLIBC_2.36' not found (required by /tmp/test/.devbox/nix/profile/default/lib/libstdc++.so.6)
  • What's happening is that that the stdenv.cc.cc.lib is expecting a GLIBC 2.36, but the code executed by the python binary (either directly, or indirectly via the matplotlib -> numpy dependencies) is linked against GLIBC 2.35.
  • I searched https://www.nixhub.io/packages/glibc and version 2.35-224 is from nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#glibc
  • So, to try fixing it I replaced stdenv.cc.cc.lib with github:nixos/nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#stdenv.cc.cc.lib

Yes it is me. I thought I could find some help from the community on it. Still I am facing other libc errors on other software installed with devbox so I try to investigate this with the clue about the libc version change you gave me.

savil commented 9 months ago

@aacebedo we've been researching a fix for your issue. In this repository https://github.com/savil/devbox-issue-1276, I have a devbox.json and some local flake.nix files that illustrate an approach to solving this problem. The README explains a bit about what its doing.

Could you please try it out in your project, and let us know if this approach works?

If it does, we can build this as a feature in Devbox. Ideally, it should work with minimal user work i.e. no need to manually write your own flake.nix.

aacebedo commented 8 months ago

@aacebedo we've been researching a fix for your issue. In this repository https://github.com/savil/devbox-issue-1276, I have a devbox.json and some local flake.nix files that illustrate an approach to solving this problem. The README explains a bit about what its doing.

Could you please try it out in your project, and let us know if this approach works?

If it does, we can build this as a feature in Devbox. Ideally, it should work with minimal user work i.e. no need to manually write your own flake.nix.

Thanks for the work ! Tried it and convert is still exhibiting the issue :(

savil commented 8 months ago

@aacebedo That's a bummer. Sorry to hear that. A couple of requests:

Request 1: repro code I thought we were running the same code as you, so am surprised it is not working. It would help if you could share a github repository with the project code you are trying to run, which exhibits this issue. That would help us debug the exact code you're running.

Request 2: try Devbox 0.8.0-dev's glibc_patch feature In the upcoming Devbox release (release notes), @gcurtis implemented this approach (with some details modified) as a product feature.

It would be great if you could try that out, and let us know if it works or not for you. Thanks!

aacebedo commented 8 months ago

@aacebedo That's a bummer. Sorry to hear that. A couple of requests:

Request 1: repro code I thought we were running the same code as you, so am surprised it is not working. It would help if you could share a github repository with the project code you are trying to run, which exhibits this issue. That would help us debug the exact code you're running.

Request 2: try Devbox 0.8.0-dev's glibc_patch feature In the upcoming Devbox release (release notes), @gcurtis implemented this approach (with some details modified) as a product feature.

  • You can use this version of devbox by setting export DEVBOX_USE_VERSION=0.8.0-dev.
  • You'll need to modify packages in devbox.json to a be json map. And set patch_glibc: true for python and librsvg. Refer to the PR Summary of as an example.
  • You'll also need to initialize the python virtualenv as shown in the virtenv script in that PR's example.

It would be great if you could try that out, and let us know if it works or not for you. Thanks!

I cloned the devbox-issue-1276 repo and ran devbox shell in it. Then I ran convert and saw the issue :( I am running devbox 0.7.1 but I'll give a try to the upcoming release as you adviced.

savil commented 8 months ago

I cloned the devbox-issue-1276 repo and ran devbox shell in it. Then I ran convert and saw the issue :(

@aacebedo 😱 oh wow. That is odd. Its been a while, but IIRC I'd been running the Dockerfile in that repo to test it out on a debian-slim image (since I have a Mac; this issue affects linux). I'll have to retry my steps in case something changed so that it is no longer working.

savil commented 8 months ago

@aacebedo sorry for the delay. After trying out the steps, I think I understand the problem. In my prototype, I've patched python and librsvg using the local flake. Those do work.

You are trying convert that came from the imagemagick nix package. You'll need to patch that via a local flake (as I illustrated with python or librsvg). We don't patch all packages automatically to be conservative, as I explain at the end of the README of https://github.com/savil/devbox-issue-1276, and until we get some validating feedback from real world projects.

Or better, I'd suggest using Devbox 0.8.0-dev and modifying the devbox.json to specify patch_glibc:true for whichever packages are hitting the libc errors. Let us know if that works, and thanks for pursuing this with us!

aacebedo commented 8 months ago

@savil thanks a lot to take a look to this !

Ok I'll try the 0.8.0-dev to check this out

aacebedo commented 8 months ago

@aacebedo sorry for the delay. After trying out the steps, I think I understand the problem. In my prototype, I've patched python and librsvg using the local flake. Those do work.

You are trying convert that came from the imagemagick nix package. You'll need to patch that via a local flake (as I illustrated with python or librsvg). We don't patch all packages automatically to be conservative, as I explain at the end of the README of https://github.com/savil/devbox-issue-1276, and until we get some validating feedback from real world projects.

Or better, I'd suggest using Devbox 0.8.0-dev and modifying the devbox.json to specify patch_glibc:true for whichever packages are hitting the libc errors. Let us know if that works, and thanks for pursuing this with us!

Ok, so I checked out the v0.8.0-dev, built it (go build ./cmd/devbox), ran a sudo nix-collect-garbage -d. I exported DEVBOX_USE_VERSION=0.8.0-dev and finally did a devbox shell with the following file:

{
  "packages": {
    "imagemagick": {
      "version": "6.9.11-60",
      "patch_glibc": true
    },
    "librsvg": {
      "version": "2.52.5"
    }
  }
}

and convert still does not work :(

convert: /nix/store/3wa1xwnfv8ada1za1r8m4vmsiz1jifqq-glibc-2.32-35/lib/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/devbox-issue-1276/.devbox/nix/profile/default/lib/librsvg-2.so.2)
convert: /nix/store/3wa1xwnfv8ada1za1r8m4vmsiz1jifqq-glibc-2.32-35/lib/libc.so.6: version `GLIBC_2.33' not found (required by /nix/store/nw2gv9l8lzi4scbddbg1vfvivzcpvag3-harfbuzz-3.2.0/lib/libharfbuzz.so.0)
savil commented 8 months ago

@aacebedo indeed, I think something is not working correctly. I can repro the error you are seeing. We're looking into it...

savil commented 8 months ago

@aacebedo thank you ! @gcurtis and I just looked at this. We added patch_glibc: true to librsvg.

This exposed a couple of issues:

  1. A bug in the current implementation. The patch_glibc was only working for one package. The fix for that is straightforward.
  2. We still do run into an error executing convert. The error has to do with a libpango not finding a g_memdup2 symbol. The root cause of that appears to be that there's a version mismatch with a different library called glib (https://gitlab.gnome.org/GNOME/glib/). We need to investigate a good fix for this, but sadly it'll take a bit of time :(. Will circle back when we have some more news.
aacebedo commented 8 months ago

@aacebedo thank you ! @gcurtis and I just looked at this. We added patch_glibc: true to librsvg.

This exposed a couple of issues:

  1. A bug in the current implementation. The patch_glibc was only working for one package. The fix for that is straightforward.
  2. We still do run into an error executing convert. The error has to do with a libpango not finding a g_memdup2 symbol. The root cause of that appears to be that there's a version mismatch with a different library called glib (https://gitlab.gnome.org/GNOME/glib/). We need to investigate a good fix for this, but sadly it'll take a bit of time :(. Will circle back when we have some more news.

Thanks to you both looking into this ! I'll continue to watch the issue and test fixes if it can help to solve this issue.

sandangel commented 5 months ago

I have the same issue. I was using poetry command after installing it with devbox add poetry. Before devbox, when using nix flake, I had to wrap poetry with this:

          poetry = pkgs.symlinkJoin {
            name = "poetry";
            paths = [ pkgs.poetry ];
            buildInputs = [ pkgs.makeWrapper ];
            postBuild = ''
              wrapProgram $out/bin/poetry --set LD_LIBRARY_PATH ${lib.makeLibraryPath [ pksgs.stdenv.cc.cc.lib ]}
            '';
          };
          ciDeps = with pkgs; [
            cmake
          ] ++ [ poetry ];
augustebaum commented 1 month ago

Same problem here. Unsetting LD_LIBRARY_PATH does not solve the issue; neither does devbox add stdenv.cc.cc.lib (or any combination of the two). I also tried managing my Python dependencies by running devbox add python@3.11 and following the instructions (installing packages with pip instead of poetry).

EDIT: Solved by setting LD_LIBRARY_PATH to <my-project-path>/.devbox/nix/profile/default/lib/.

EDIT: Not quite solved: My script now works, but:

  1. python3 -c "import numpy" fails with: 
    
(outerbounds-examples-py3.11) ❯ p3 -c 'import numpy'
Traceback (most recent call last):
File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/core/__init__.py", line 24, in <module>
from . import multiarray
File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/core/multiarray.py", line 10, in <module>
from . import overrides
File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/core/overrides.py", line 8, in <module>
from numpy.core._multiarray_umath import (
ImportError: libz.so.1: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/init.py", line 130, in from numpy.config import show as show_config File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/config.py", line 4, in from numpy.core._multiarray_umath import ( File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/core/init.py", line 50, in raise ImportError(msg) ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed.

We have compiled some common reasons and troubleshooting tips at:

https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help.

Original error was: libz.so.1: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "", line 1, in File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/init.py", line 135, in raise ImportError(msg) from e ImportError: Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python interpreter from there.

although I think it might've failed even before the fix.

2. All `nix` commands now fail with

nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version GLIBCXX_3.4.31' not found (required by nix) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: versionGLIBCXX_3.4.32' not found (required by nix) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixexpr.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: versionGLIBCXX_3.4.31' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixexpr.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version GLIBCXX_3.4.31' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixmain.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: versionGLIBCXX_3.4.32' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixmain.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixfetchers.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: versionGLIBCXX_3.4.31' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixfetchers.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version GLIBCXX_3.4.31' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixstore.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: versionGLIBCXX_3.4.32' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixstore.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version GLIBCXX_3.4.31' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixutil.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: versionGLIBCXX_3.4.32' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixutil.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixcmd.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: versionGLIBCXX_3.4.31' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixcmd.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by /nix/store/dgc42gd76af2syp7fk5d9mv2abqrggkg-aws-sdk-cpp-1.11.207/lib/libaws-cpp-sdk-transfer.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: versionGLIBCXX_3.4.32' not found (required by /nix/store/dgc42gd76af2syp7fk5d9mv2abqrggkg-aws-sdk-cpp-1.11.207/lib/libaws-cpp-sdk-s3.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by /nix/store/dgc42gd76af2syp7fk5d9mv2abqrggkg-aws-sdk-cpp-1.11.207/lib/libaws-cpp-sdk-core.so) nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: versionGLIBCXX_3.4.32' not found (required by /nix/store/a88l22177h3mb4ifiy734mr71c40pfxp-aws-crt-cpp-0.24.7/lib/libaws-crt-cpp.so)



EDIT: Importing numpy does not fail when done in IPython (installed through devbox). Uh, what?
augustebaum commented 1 month ago

I ended up using a shell.nix instead, using nixpkgs unstable rather than 23.11, and that seemed to work alright. I'll try devbox again when I have the time.