Open saippua opened 1 month ago
grpc
for python3Packages.tensorflow
gets overriden in pkgs/top-level/python-packages.nix:14946
.
Ok, so grpcio
is overridden because grpc
and protobuf
need to be overridden. grpc
is overridden because of some conflict with nvcc
and protobuf
is overridden for some reason concerning abseil
.
Is there a standard procedure for cases like this? I guess applying the tensorflow overrides to grpcio
, grpc
and protobuf
globally within my nixpkgs would solve the collision, assuming that the overrides don't spawn other errors with other packages, but that doesn't seem like a very stable solution. Is there some way to let the colliding packages coexist within the environment?
This is a duplicate of https://github.com/NixOS/nixpkgs/issues/323965 and has been a pain point for months, see https://github.com/NixOS/nixpkgs/pull/286125#issuecomment-1956370973 and https://github.com/NixOS/nixpkgs/pull/266467#discussion_r1546203656.
Replacing tensorflow
with tensorflow-bin
suffices. This is probably the simplest and easiest solution.
I guess applying the tensorflow overrides to
grpcio
,grpc
andprotobuf
globally within my nixpkgs would solve the collision, assuming that the overrides don't spawn other errors with other packages, but that doesn't seem like a very stable solution.
If you really want to do a source build, then the simplest way is to override torch with the tensorflow overrides, which has the advantage of only modifying torch, but has the disadvantage of manually finding the conflicting dependencies.
A flake is given below. Note that this requires recompiling torch (but not tensorflow) which took ~50 minutes on my laptop.
{
inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
outputs = { self, nixpkgs }:
let
system = "x86_64-linux";
pkgs = nixpkgs.legacyPackages.${system};
# see tensorflow-build in pkgs/top-level/python-packages.nix
self = pkgs.python311Packages;
compat = rec {
protobufTF = pkgs.protobuf_21.override {
abseil-cpp = pkgs.abseil-cpp_202301;
};
grpcTF = (pkgs.grpc.overrideAttrs (
oldAttrs: rec {
version = "1.27.3";
src = pkgs.fetchFromGitHub {
owner = "grpc";
repo = "grpc";
rev = "v${version}";
hash = "sha256-PpiOT4ZJe1uMp5j+ReQulC9jpT0xoR2sAl6vRYKA0AA=";
fetchSubmodules = true;
};
patches = [ ];
postPatch = ''
sed -i "s/-std=c++11/-std=c++17/" CMakeLists.txt
echo "set(CMAKE_CXX_STANDARD 17)" >> CMakeLists.txt
'';
}
)
).override {
protobuf = protobufTF;
};
protobuf-pythonTF = self.protobuf.override {
protobuf = protobufTF;
};
grpcioTF = self.grpcio.override {
protobuf = protobufTF;
grpc = grpcTF;
};
tensorboard-plugin-profileTF = self.tensorboard-plugin-profile.override {
protobuf = protobuf-pythonTF;
};
tensorboardTF = self.tensorboard.override {
grpcio = grpcioTF;
protobuf = protobuf-pythonTF;
tensorboard-plugin-profile = tensorboard-plugin-profileTF;
};
};
torch = self.torch.override {
tensorboard = compat.tensorboardTF;
protobuf = compat.protobuf-pythonTF;
};
in
{
devShell.${system} = pkgs.mkShell {
packages = [
(pkgs.python311.withPackages (ps: [
ps.tensorflow
torch
]))
];
};
};
}
A more sophisticated approach uses pkgs.python311Packages.overrideScope
to create an alternative universe with the tensorflow overrides (this is what I do to package gpjax in maipkgs for example). I reverted the modifications to google-auth
to maintain exactly the same behavior as the previous flake; although google-auth
only uses grpcio
in nativeCheckInputs
, by default nix is input-addressed rather than content-addressed, so this changes the hash and therefore causes rebuilds. The advantage of this flake is that tensorflow's overrides are automatically propagated to every python package while being localized to python3Packages'
, i.e., much more fine-grain than a nixpkgs override.
{
inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
outputs = { self, nixpkgs }:
let
system = "x86_64-linux";
pkgs = nixpkgs.legacyPackages.${system};
# see tensorflow-build in pkgs/top-level/python-packages.nix
compat = rec {
protobuf = pkgs.protobuf_21.override {
abseil-cpp = pkgs.abseil-cpp_202301;
};
grpc = (pkgs.grpc.overrideAttrs (
oldAttrs: rec {
version = "1.27.3";
src = pkgs.fetchFromGitHub {
owner = "grpc";
repo = "grpc";
rev = "v${version}";
hash = "sha256-PpiOT4ZJe1uMp5j+ReQulC9jpT0xoR2sAl6vRYKA0AA=";
fetchSubmodules = true;
};
patches = [ ];
postPatch = ''
sed -i "s/-std=c++11/-std=c++17/" CMakeLists.txt
echo "set(CMAKE_CXX_STANDARD 17)" >> CMakeLists.txt
'';
}
)
).override {
inherit protobuf;
};
};
python3Packages' = pkgs.python311Packages.overrideScope (final: prev: {
protobuf = prev.protobuf.override {
inherit (compat) protobuf;
};
grpcio = prev.grpcio.override {
inherit (compat) protobuf grpc;
};
# probably unnecessary
google-auth = prev.google-auth.override {
grpcio = prev.grpcio.override {
inherit (prev) protobuf;
};
};
});
python' = python3Packages'.python.override {
packageOverrides = final: prev: python3Packages';
self = python';
};
in
{
devShell.${system} = pkgs.mkShell {
packages = [
(python'.withPackages (ps: with ps; [
tensorflow
torch
]))
];
};
};
}
Unfortunately both flakes require recompiling torch which is quite expensive. In terms of a nixpkgs PR, we should probably export tensorboardTF
as what is currently called tensorboard
(right now tensorflow
and tensorboard
conflict, which is quite surprising). That itself is not sufficient, as e.g. orbax-checkpoint
conflicts through protobuf
. I can only think of exposing the *TF
packages in compat
at the top level and manually overwriting machine learning packages that are commonly used with tensorflow, but this is obviously not ideal.
Is there some way to let the colliding packages coexist within the environment?
Not as far as I know; apparently this is an inherent limitation of python. The manual says
We also cannot package multiple versions of the same package since this may cause conflicts in
PYTHONPATH
.
Steps To Reproduce
Here is a flake for reproduction:
nix develop
to run.Build log
I also tried running the flake on
nixpkgs-unstable
. This also fails, but the error is slightly different:Additional context
I'm trying to create a shell with
tensorflow
andpytorch
, however, I get the collision error. The same error happens in a shell with justtensorflow
andgrpcio
, so I'm assuming that means that Tensorflow is using some modified version ofgrpcio
which collides with thenixpkgs
version.I tried to debug this myself, but after looking at the
tensorflow
derivation, I couldn't find anything which looked like it could be overriding thegrpcio
...However, I can have a shell with
torch
andgrpcio
and it works just fine, so I'm assuming this is a Tensorflow issue?Notify maintainers
@abbradar @ndl @mweinelt
Metadata
Add a :+1: reaction to issues you find important.