mweinelt
commented
1 month ago grpc for python3Packages.tensorflow gets overriden in pkgs/top-level/python-packages.nix:14946.
Open saippua opened 1 month ago
mweinelt
commented
1 month ago grpc for python3Packages.tensorflow gets overriden in pkgs/top-level/python-packages.nix:14946.
saippua
commented
1 month ago Ok, so grpcio is overridden because grpc and protobuf need to be overridden. grpc is overridden because of some conflict with nvcc and protobuf is overridden for some reason concerning abseil.
Is there a standard procedure for cases like this? I guess applying the tensorflow overrides to grpcio, grpc and protobuf globally within my nixpkgs would solve the collision, assuming that the overrides don't spawn other errors with other packages, but that doesn't seem like a very stable solution. Is there some way to let the colliding packages coexist within the environment?
stephen-huan
commented
1 month ago This is a duplicate of https://github.com/NixOS/nixpkgs/issues/323965 and has been a pain point for months, see https://github.com/NixOS/nixpkgs/pull/286125#issuecomment-1956370973 and https://github.com/NixOS/nixpkgs/pull/266467#discussion_r1546203656.
Replacing tensorflow with tensorflow-bin suffices. This is probably the simplest and easiest solution.
I guess applying the tensorflow overrides to
grpcio,grpcandprotobufglobally within my nixpkgs would solve the collision, assuming that the overrides don't spawn other errors with other packages, but that doesn't seem like a very stable solution.
If you really want to do a source build, then the simplest way is to override torch with the tensorflow overrides, which has the advantage of only modifying torch, but has the disadvantage of manually finding the conflicting dependencies.
A flake is given below. Note that this requires recompiling torch (but not tensorflow) which took ~50 minutes on my laptop.
{
inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
outputs = { self, nixpkgs }:
let
system = "x86_64-linux";
pkgs = nixpkgs.legacyPackages.${system};
# see tensorflow-build in pkgs/top-level/python-packages.nix
self = pkgs.python311Packages;
compat = rec {
protobufTF = pkgs.protobuf_21.override {
abseil-cpp = pkgs.abseil-cpp_202301;
};
grpcTF = (pkgs.grpc.overrideAttrs (
oldAttrs: rec {
version = "1.27.3";
src = pkgs.fetchFromGitHub {
owner = "grpc";
repo = "grpc";
rev = "v${version}";
hash = "sha256-PpiOT4ZJe1uMp5j+ReQulC9jpT0xoR2sAl6vRYKA0AA=";
fetchSubmodules = true;
};
patches = [ ];
postPatch = ''
sed -i "s/-std=c++11/-std=c++17/" CMakeLists.txt
echo "set(CMAKE_CXX_STANDARD 17)" >> CMakeLists.txt
'';
}
)
).override {
protobuf = protobufTF;
};
protobuf-pythonTF = self.protobuf.override {
protobuf = protobufTF;
};
grpcioTF = self.grpcio.override {
protobuf = protobufTF;
grpc = grpcTF;
};
tensorboard-plugin-profileTF = self.tensorboard-plugin-profile.override {
protobuf = protobuf-pythonTF;
};
tensorboardTF = self.tensorboard.override {
grpcio = grpcioTF;
protobuf = protobuf-pythonTF;
tensorboard-plugin-profile = tensorboard-plugin-profileTF;
};
};
torch = self.torch.override {
tensorboard = compat.tensorboardTF;
protobuf = compat.protobuf-pythonTF;
};
in
{
devShell.${system} = pkgs.mkShell {
packages = [
(pkgs.python311.withPackages (ps: [
ps.tensorflow
torch
]))
];
};
};
}A more sophisticated approach uses pkgs.python311Packages.overrideScope to create an alternative universe with the tensorflow overrides (this is what I do to package gpjax in maipkgs for example). I reverted the modifications to google-auth to maintain exactly the same behavior as the previous flake; although google-auth only uses grpcio in nativeCheckInputs, by default nix is input-addressed rather than content-addressed, so this changes the hash and therefore causes rebuilds. The advantage of this flake is that tensorflow's overrides are automatically propagated to every python package while being localized to python3Packages', i.e., much more fine-grain than a nixpkgs override.
{
inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
outputs = { self, nixpkgs }:
let
system = "x86_64-linux";
pkgs = nixpkgs.legacyPackages.${system};
# see tensorflow-build in pkgs/top-level/python-packages.nix
compat = rec {
protobuf = pkgs.protobuf_21.override {
abseil-cpp = pkgs.abseil-cpp_202301;
};
grpc = (pkgs.grpc.overrideAttrs (
oldAttrs: rec {
version = "1.27.3";
src = pkgs.fetchFromGitHub {
owner = "grpc";
repo = "grpc";
rev = "v${version}";
hash = "sha256-PpiOT4ZJe1uMp5j+ReQulC9jpT0xoR2sAl6vRYKA0AA=";
fetchSubmodules = true;
};
patches = [ ];
postPatch = ''
sed -i "s/-std=c++11/-std=c++17/" CMakeLists.txt
echo "set(CMAKE_CXX_STANDARD 17)" >> CMakeLists.txt
'';
}
)
).override {
inherit protobuf;
};
};
python3Packages' = pkgs.python311Packages.overrideScope (final: prev: {
protobuf = prev.protobuf.override {
inherit (compat) protobuf;
};
grpcio = prev.grpcio.override {
inherit (compat) protobuf grpc;
};
# probably unnecessary
google-auth = prev.google-auth.override {
grpcio = prev.grpcio.override {
inherit (prev) protobuf;
};
};
});
python' = python3Packages'.python.override {
packageOverrides = final: prev: python3Packages';
self = python';
};
in
{
devShell.${system} = pkgs.mkShell {
packages = [
(python'.withPackages (ps: with ps; [
tensorflow
torch
]))
];
};
};
}Unfortunately both flakes require recompiling torch which is quite expensive. In terms of a nixpkgs PR, we should probably export tensorboardTF as what is currently called tensorboard (right now tensorflow and tensorboard conflict, which is quite surprising). That itself is not sufficient, as e.g. orbax-checkpoint conflicts through protobuf. I can only think of exposing the *TF packages in compat at the top level and manually overwriting machine learning packages that are commonly used with tensorflow, but this is obviously not ideal.
Is there some way to let the colliding packages coexist within the environment?
Not as far as I know; apparently this is an inherent limitation of python. The manual says
We also cannot package multiple versions of the same package since this may cause conflicts in
PYTHONPATH.
Steps To Reproduce
Here is a flake for reproduction:
nix developto run.Build log
I also tried running the flake on
nixpkgs-unstable. This also fails, but the error is slightly different:Additional context
I'm trying to create a shell with
tensorflowandpytorch, however, I get the collision error. The same error happens in a shell with justtensorflowandgrpcio, so I'm assuming that means that Tensorflow is using some modified version ofgrpciowhich collides with thenixpkgsversion.I tried to debug this myself, but after looking at the
tensorflowderivation, I couldn't find anything which looked like it could be overriding thegrpcio...However, I can have a shell with
torchandgrpcioand it works just fine, so I'm assuming this is a Tensorflow issue?Notify maintainers
@abbradar @ndl @mweinelt
Metadata
Add a :+1: reaction to issues you find important.