Open richardcooper opened 2 years ago
pip enables to define extra lib/header directory using environment variables. These packages are installed in the DEVSHELL_DIR directory. This should work:
[devshell]
packages = [
"python39",
"python39Packages.setuptools",
"libmemcached",
"cyrus_sasl.dev",
"zlib.dev",
"zlib",
"gcc",
]
[[env]]
name = "CPPFLAGS"
eval = "-I${DEVSHELL_DIR}/include"
[[env]]
name = "LDFLAGS"
eval = "-L${DEVSHELL_DIR}/lib"
Note that we need to set the dev output for zlib.dev
and cyrus_sasl.dev
as packages
just takes the default output.
We could ease that process and define an extra language python
in devshell to enable
this kind of C dependencies for pip (similar to what we do in c.nix).
We would end up with something like this:
imports = [
"language.python"
]
language.python.includes = [
"zlib",
"cyrus_sasl",
]
language.python.libraries = [
"zlib",
"libmemcached"
]
[devshell]
packages = [
"python39",
"python39Packages.setuptools",
]
FTR I was having the same problem and I also had to patch the $LD_LIBRARY_PATH
env var. Also, I was writing the configuration directly in nix, so I had to rewrite those ${DEVSHELL_DIR}
into $DEVSHELL_DIR
to avoid a failure. My intention was to use psycopg2
. A code snippet:
pkgs.devshell.mkShell {
env = [
# Disponibilize compile headers and dependencies
{name="CPPFLAGS"; eval = "-I$DEVSHELL_DIR/include";}
{name="LDFLAGS"; eval = "-L$DEVSHELL_DIR/lib";}
{name="LD_LIBRARY_PATH"; eval = "$DEVSHELL_DIR/lib:$LD_LIBRARY_PATH";}
];
packages = with pkgs; [
# Build dependencies to allow `poetry install`
openssl.dev
postgresql.lib
];
};
If I didn't patch $LD_LIBRARY_PATH
, I'd get this error:
> python -c 'import psycopg2'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/var/home/yajo/prodevel/example/.venv/lib/python3.8/site-packages/psycopg2/__init__.py", line 51, in <module>
from psycopg2._psycopg import ( # noqa
ImportError: libpq.so.5: cannot open shared object file: No such file or directory
I'm using the same techniques outlined above and am still having issues installing psycopg2 via pip (is that the problem?). My config:
# https://numtide.github.io/devshell
[[commands]]
package = "devshell.cli"
help = "Per project developer environments"
[devshell]
packages = [
"gcc",
"openssl",
"openssl.dev",
"postgresql",
# Python
"python310Full",
"python310Packages.setuptools",
]
[[env]]
name = "CPPFLAGS"
eval = "-I${DEVSHELL_DIR}/include"
[[env]]
name = "LDFLAGS"
eval = "-L${DEVSHELL_DIR}/lib"
[[env]]
name = "LD_LIBRARY_PATH"
eval = "${DEVSHELL_DIR}/lib:${LD_LIBRARY_PATH}"
but I receive the following when I try to build my venv.
In file included from psycopg/win32_support.c:28:
In file included from ./psycopg/psycopg.h:38:
./psycopg/config.h:69:13: warning: unused function 'Dprintf' [-Wunused-function]
static void Dprintf(const char *fmt, ...)
^
1 warning generated.
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/7cmwkhhrhnhwz5bwm3g76df2xa69mr26-devshell-dir/include "-DPSYCOPG_VERSION=2.9.3 (dt dec pq3 ext lo64)" -DPSYCOPG_DEBUG=1 -DPG_VERSION_NUM=140004 -DHAVE_LO64=1 -DPSYCOPG_DEBUG=1 -I/Volumes/Work/erezlife/venv/include -I/nix/store/6n0zvdpam247lq10xfrp35fqmnmkikzx-python3-3.10.5/include/python3.10 -I. -I/nix/store/7cmwkhhrhnhwz5bwm3g76df2xa69mr26-devshell-dir/include -I/nix/store/7cmwkhhrhnhwz5bwm3g76df2xa69mr26-devshell-dir/include/server -I/nix/store/1k3qq40cd22vl8ily5l9ck9n5fi37czn-icu4c-71.1-dev/include -I/nix/store/lg2lxrbwk2gbm74n4kqnqc6jq4326xzr-libxml2-2.9.14-dev/include/libxml2 -I/nix/store/qbynak9nki5ashg1rz5gx8mlf2r8hcs4-lz4-1.9.3-dev/include -c psycopg/xid_type.c -o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/xid_type.o
clang -bundle -undefined dynamic_lookup -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.12/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.4.8/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.5/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.39.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-6.3p08/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.3-p20220507/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.1.1q/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-tcl-8.6.11/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-tk-8.6.11/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libX11-1.7.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xorgproto-2021.5/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib -L/nix/store/869c76qhfn4jgbwiqnf6sw5jcc35lv28-tzdata-2022a/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.12/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.4.8/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.5/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.39.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-6.3p08/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.3-p20220507/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.1.1q/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-tcl-8.6.11/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-tk-8.6.11/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libX11-1.7.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xorgproto-2021.5/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib -L/nix/store/869c76qhfn4jgbwiqnf6sw5jcc35lv28-tzdata-2022a/lib -L/nix/store/7cmwkhhrhnhwz5bwm3g76df2xa69mr26-devshell-dir/lib -I/nix/store/7cmwkhhrhnhwz5bwm3g76df2xa69mr26-devshell-dir/include build/temp.macosx-10.12-x86_64-cpython-310/psycopg/adapter_asis.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/adapter_binary.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/adapter_datetime.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/adapter_list.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/adapter_pboolean.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/adapter_pdecimal.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/adapter_pfloat.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/adapter_pint.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/adapter_qstring.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/aix_support.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/bytes_format.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/column_type.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/connection_int.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/connection_type.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/conninfo_type.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/cursor_int.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/cursor_type.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/diagnostics_type.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/error_type.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/green.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/libpq_support.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/lobject_int.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/lobject_type.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/microprotocols.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/microprotocols_proto.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/notify_type.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/pqpath.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/psycopgmodule.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/replication_connection_type.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/replication_cursor_type.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/replication_message_type.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/solaris_support.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/typecast.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/utils.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/win32_support.o build/temp.macosx-10.12-x86_64-cpython-310/psycopg/xid_type.o -L/nix/store/6n0zvdpam247lq10xfrp35fqmnmkikzx-python3-3.10.5/lib -L/nix/store/9zzdsi078dfvwk3c8mp0ah4n11qdhbnj-postgresql-14.4-lib/lib -L/nix/store/2bsx47y6a98pwc2dhhva34a8nw4hk4f6-libxml2-2.9.14/lib -L/nix/store/z90ch0v4wddigb5q4vh7r1xz8bffqic5-lz4-1.9.3/lib -lpq -lssl -lcrypto -o build/lib.macosx-10.12-x86_64-cpython-310/psycopg2/_psycopg.cpython-310-darwin.so
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.12/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.4.8/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.5/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.39.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-6.3p08/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.3-p20220507/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.1.1q/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-tcl-8.6.11/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-tk-8.6.11/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libX11-1.7.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xorgproto-2021.5/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib'
ld: warning: directory not found for option '-L/nix/store/869c76qhfn4jgbwiqnf6sw5jcc35lv28-tzdata-2022a/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.12/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.4.8/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.5/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.39.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-6.3p08/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.3-p20220507/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.1.1q/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-tcl-8.6.11/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-tk-8.6.11/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libX11-1.7.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xorgproto-2021.5/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib'
ld: warning: directory not found for option '-L/nix/store/869c76qhfn4jgbwiqnf6sw5jcc35lv28-tzdata-2022a/lib'
ld: library not found for -lssl
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/usr/bin/clang' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> psycopg2
You seem to be missing postgresql.lib
. Check out https://github.com/numtide/devshell/issues/172#issuecomment-1094675420.
Ah missed that, thanks!
Describe the bug
Using a Python venv in a devshell it is not possible to install pylbmc, which is the Python wrapper around
libmemcached
. I've installed the dependencies ofpylibmc
(libmemcached
&zlib
) in the devshell, but thepylibmc
installer cannot find thelibmemcached
headers.To Reproduce
Steps to reproduce the behavior:
nix flake new -t "github:numtide/devshell" pylibmc_issue
cd pylibmc_issue
devshell.toml
to have the the following contentsnix develop
python -m venv venv
source venv/bin/activate
pip install pylibmc
The result is
Expected behavior
I would expect
pylibmc
to install without error, as it does if I use Nix withoutdevshell
.Instead of using
devshell
I can use the followingflake.nix
file which installs the same packages.Using that
flake.nix
file, now steps 4-7 above succeed.System information
macOS Monterey 12.2.1 on an Intel Mac Nix: 2.6.0 Devshell rev: 7033f64
Additional context
I am aware that
nixpkgs
contains apython39Packages.pylibmc
package which I could use instead of installingpylibmc
viapip
. The problem is that ultimately I'm trying to usedevshell
on a large Python application which has many dependencies including multiple C-based packages. Many of those packages don't exist innixpkgs
so unfortunately sticking to only packages installed vianixpkgs
is not an option for me.