grpcio breaks tensorflow

[mtrsk]

And this seems to be related to this issue.

This is my current shell.nix:

{ pkgs ? import <nixpkgs> {} }:

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "refs/tags/3.0.1";
  }) {
    python = "python38";
    pkgs = pkgs;
  };

  py_env = mach-nix.mkPython rec {
    requirements = builtins.readFile ./requirements.txt;

    providers = {
      _default = "nixpkgs,wheel,sdist";
    };
  };

  dev_env = [ ((py_env).override( args:{ ignoreCollisions=true; })) ];

  system_deps = with pkgs; [
    leptonica
    python38Packages.jupyter
    tesseract4
    opencv4
  ];
in
pkgs.mkShell rec {
  buildInputs = system_deps ++ dev_env;
}

and requirements.txt:

absl-py==0.10.0
astunparse==1.6.3
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
gast==0.3.3
google-auth-oauthlib==0.4.1
google-auth==1.22.1
google-pasta==0.2.0
grpcio==1.33.1
h5py==2.10.0
idna==2.10
keras-preprocessing==1.1.2
markdown==3.3.2
numpy==1.18.5
oauthlib==3.1.0
opencv-python==4.4.0.44
opt-einsum==3.3.0
protobuf==3.13.0
pyasn1-modules==0.2.8
pyasn1==0.4.8
requests-oauthlib==1.3.0
requests==2.24.0
rsa==4.6; python_version >= "3.5"
six==1.15.0
tensorboard-plugin-wit==1.7.0
tensorboard==2.3.0
tensorflow-estimator==2.3.0
tensorflow==2.3.1
termcolor==1.1.0
tesserocr==2.5.1
urllib3==1.25.11
werkzeug==1.0.1
wrapt==1.12.1

Which causes the following error:

Mach-nix version: 3.0.1
Python: 3.8.6
Cause: Requirements conflict: Requirement.parse('grpcio==1.33.1')
The requirements which caused the error:
  grpcio==1.33.1

error: --- Error ----------------------------------------------------------------------------------- nix-shell
error: --- Error --- nix-daemon
builder for '/nix/store/ia9p8yppn7zv3s7v6m386z2g7n4132jx-mach_nix_file.drv' failed with exit code 1; last 10 log lines:
     'pip': 'nixpkgs,sdist',
     'setupmeta': 'wheel',
     'setuptools': 'nixpkgs',
     'wheel': 'nixpkgs,sdist'}
  Mach-nix version: 3.0.1
  Python: 3.8.6
  Cause: Requirements conflict: Requirement.parse('grpcio==1.33.1')
  The requirements which caused the error:
    grpcio==1.33.1

Any hints if it's possible to use match.nix to patch tensorflow or this has to done via an overlay on nixpkgs?

[DavHau]

Hey the problem is unrelated the nixpkgs issue you linked. In your case mach-nix failed while resolving the requirements, it did not yet start building stuff.

Your requirements.txt simply seems to be invalid or not resolvable. You can probably fix that by removing some restrictions. For example replace grpcio==1.33.1 with grpcio. Or better remove grpcio from the requirements at all, because it seems to be already specified by some other top level dependency (hence the conflict). In general, stating top level requirements only instead of flat requirements should be preferred. See https://github.com/DavHau/mach-nix/issues/140

Despite it not being the problem here, patching can be done for example by using simplified overrides Example: _.tensorflow.patches = [ ./your.patch ] or _.tensorflow.patches.add = [ ./your.patch ]

[mtrsk]

Thanks, I'll try your solution. This requirements.txt was generated via poetry export --without-hashes -f requirements.txt -o requirements-dev.txt, from a poetry-based project.

[mtrsk]

Hey the problem is unrelated the nixpkgs issue you linked. In your case mach-nix failed while resolving the requirements, it did not yet start building stuff.

Your requirements.txt simply seems to be invalid or not resolvable. You can probably fix that by removing some restrictions. For example replace grpcio==1.33.1 with grpcio. Or better remove grpcio from the requirements at all, because it seems to be already specified by some other top level dependency (hence the conflict). In general, stating top level requirements only instead of flat requirements should be preferred. See #140

Despite it not being the problem here, patching can be done for example by using simplified overrides Example: _.tensorflow.patches = [ ./your.patch ] or _.tensorflow.patches.add = [ ./your.patch ]

It worked, thanks!

I'm gonna close this issue, I'm currently trying match.nix in some work projects to see if I can introduce nix there.

[DavHau]

I'm gonna close this issue, I'm currently trying match.nix in some work projects to see if I can introduce nix there.

Sounds exciting. Let me know if there are any more problems or missing features.