DavHau / mach-nix

Create highly reproducible python environments
MIT License
862 stars 106 forks source link

resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=Requirement.parse('dask==2.22.0'), parent=None) #66

Closed InLaw closed 4 years ago

InLaw commented 4 years ago
let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "refs/tags/2.2.1";
  });
in mach-nix.buildPythonPackage rec {
  pname = "deeplabcut";
  version = "v2.2bb7";
  pkgs = (import <unstable> {});
  src = builtins.fetchGit{
    url = "https://github.com/DeepLabCut/DeepLabCut";
    ref = "master";
    rev = "c98f0d308636be5c4feaea4a35e77c2d974aa710";
  };
  requirements = ''
    affine==2.3.0
appdirs==1.4.4
#appnope==0.1.0; python_version >= "3.3" and sys_platform == "darwin" or platform_system == "Darwin"
arcgis==1.8.2
area==1.1.1
argon2-cffi==20.1.0
#atomicwrites==1.4.0; sys_platform == "win32"
attrs==19.3.0
autoflake==1.3.1
backcall==0.2.0
beautifulsoup4==4.9.1
black==19.10b0
bleach==3.1.5
bokeh==2.1.1
certifi==2020.6.20
cffi==1.14.1
chardet==3.0.4
click==7.1.2
click-plugins==1.1.1
cligj==0.5.0
#colorama==0.4.3; python_version >= "3.3" and sys_platform == "win32" or sys_platform == "win32"
colorcet==2.0.2
cryptography==3.0
cycler==0.10.0
dask==2.22.0
datashader==0.10.0
datashape==0.5.2
decorator==4.4.2
defusedxml==0.6.0
descartes==1.1.0
entrypoints==0.3
feather-format==0.4.1
fiona==1.8.13.post1
flake8==3.8.3
flufl.enum==4.1.1
geoalchemy2==0.8.4
geofeather==0.3.0
geographiclib==1.50
geojson==2.5.0
geojsonio==0.0.3
geolinks==0.2.0
geopandas==0.8.1
geopy==2.0.0
github3.py==1.3.0
googledrivedownloader==0.4
greedy==0.1.1
idna==2.10
imageio==2.9.0
importlib-metadata==1.7.0
ipykernel==5.3.4
ipython==7.17.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.17.2
jeepney==0.4.3
jinja2==2.11.2
joblib==0.16.0
json5==0.9.5
jsonschema==3.2.0
jupyter-client==6.1.6
jupyter-core==4.6.3
jupyterlab==2.2.4
jupyterlab-server==1.2.0
jwcrypto==0.7
keyring==21.3.0
kiwisolver==1.2.0
lerc==0.1.0
libpysal==4.3.0
lidar==0.5.0
llvmlite==0.34.0rc1
lxml==4.5.2
markupsafe==1.1.1
matplotlib==3.3.0
mccabe==0.6.1
mercantile==1.1.5
missingno==0.4.2
mistune==0.8.4
momepy==0.3.0
more-itertools==8.4.0
multipledispatch==0.6.0
munch==2.5.0
mypy==0.782
mypy-extensions==0.4.3
nbconvert==5.6.1
nbformat==5.0.7
networkx==2.4
notebook==6.1.1
ntlm-auth==1.5.0
numba==0.50.1
numpy==1.19.1
oauthlib==3.1.0
osm-diff-tool==1.1.0
osmapi==1.2.2
osmnet==0.1.6
osmpythontools==0.2.8
overpass==0.7
overpy2==0.4.2
owslib==0.20.0
packaging==20.4
pandas==1.1.0
pandocfilters==1.4.2
param==1.9.3
parso==0.7.1
pathlib==1.0.1
pathspec==0.8.0
pexpect==4.8.0
pickleshare==0.7.5
pillow==7.2.0
pluggy==0.13.1
pooch==1.1.1
prometheus-client==0.8.0
prompt-toolkit==3.0.5
ptyprocess==0.6.0
py==1.9.0
pyarrow==1.0.0
pycodestyle==2.6.0
#pycparser==2.20; platform_system == "Windows" and python_version != "3.3" or platform_system == "Windows" and python_version != "3.3" and sys_platform == "linux"
pycsw==2.4.2
pyct==0.4.6
pydal==20200714.1
pyflakes==2.2.0
pygeos==0.7.1
pygis==0.1.3
pygments==2.6.1
#pykerberos==1.2.1; platform_system == "Windows" and sys_platform != "win32"
pyparsing==2.4.7
pyproj==2.6.1.post1
pyrsistent==0.16.0
pyshp==2.1.0
pysimplegui==4.28.0
pytest==5.4.3
python-dateutil==2.8.1
pytz==2020.1
pywavelets==1.1.1
#pywin32==228; sys_platform == "win32" or platform_system == "Windows"
#pywin32-ctypes==0.2.0; sys_platform == "win32"
#pywinpty==0.5.7; os_name == "nt"
pywps==4.2.7
pyyaml==5.3.1
pyzmq==19.0.2
rasterio==1.1.5
rasterstats==0.15.0
regex==2020.7.14
requests==2.24.0
#requests-kerberos==0.12.0; platform_system == "Windows"
requests-ntlm==1.1.0
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
richdem==0.3.4
rio-cogeo==1.1.10
rsgis==0.0.3
scikit-image==0.17.2
scikit-learn==0.23.2
scipy==1.5.2
seaborn==0.10.1
secretstorage==3.1.2
send2trash==1.5.0
shapely==1.7.0
simplejson==3.17.2
six==1.15.0
snuggs==1.4.7
soupsieve==1.9.6
spatial==0.2.0
sqlalchemy==1.3.18
supermercado==0.1.1
terminado==0.8.3
testpath==0.4.4
threadpoolctl==2.1.0
tifffile==2020.7.24
toml==0.10.1
toolz==0.10.0
tornado==6.0.4
tqdm==4.48.2
traitlets==4.3.3
typed-ast==1.4.1
typing-extensions==3.7.4.2
ujson==3.1.0
uritemplate==3.0.1
urllib3==1.25.10
verde==1.5.0
wcwidth==0.2.5
webencodings==0.5.1
werkzeug==1.0.1
widgetsnbextension==3.5.1
#winkerberos==0.7.0; platform_system == "Windows" and sys_platform == "win32" or platform_system == "Windows"
xarray==0.16.0
xarray-spatial==0.0.8
xmltodict==0.12.0
zipp==3.1.0
  '';
}

error:

building '/nix/store/5ssy5glf78kh973w5bz5ps82q2kri50n-mach_nix_file.drv'...
Traceback (most recent call last):
Traceback (most recent call last):
  File "/nix/store/5l1ps1iaml9aivqsd2796cryadpy74py-ch8zsbs2vlfqdr6rpz85y455sfmz6vyf-source/mach_nix/generate.py", line 63, in main
    expr = generator.generate(reqs)
  File "/nix/store/5l1ps1iaml9aivqsd2796cryadpy74py-ch8zsbs2vlfqdr6rpz85y455sfmz6vyf-source/mach_nix/generators/overides_generator.py", line 34, in generate
    pkgs = self.resolver.resolve(reqs)
  File "/nix/store/5l1ps1iaml9aivqsd2796cryadpy74py-ch8zsbs2vlfqdr6rpz85y455sfmz6vyf-source/mach_nix/resolver/resolvelib_resolver.py", line 63, in resolve
    result = resolvelib.Resolver(Provider(self.nixpkgs, self.deps_provider), reporter).resolve(reqs, max_rounds=1000)
  File "/nix/store/njdwizism6z27b18dcngxlm4pw9aa07f-python3-3.7.8-env/lib/python3.7/site-packages/resolvelib/resolvers.py", line 413, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/nix/store/njdwizism6z27b18dcngxlm4pw9aa07f-python3-3.7.8-env/lib/python3.7/site-packages/resolvelib/resolvers.py", line 319, in resolve
    raise ResolutionImpossible(causes)
resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=Requirement.parse('dask==2.22.0'), parent=None), RequirementInformation(requirement=Requirement.parse('dask[complete]>=0.18.0'), parent=Candidate(name='datashader', ver=<Version('0.10.0')>, extras=()))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/5l1ps1iaml9aivqsd2796cryadpy74py-ch8zsbs2vlfqdr6rpz85y455sfmz6vyf-source/mach_nix/generate.py", line 94, in <module>
    main()
  File "/nix/store/5l1ps1iaml9aivqsd2796cryadpy74py-ch8zsbs2vlfqdr6rpz85y455sfmz6vyf-source/mach_nix/generate.py", line 65, in main
    handle_resolution_impossible(e, requirements, providers_json, py_ver_str)
  File "/nix/store/5l1ps1iaml9aivqsd2796cryadpy74py-ch8zsbs2vlfqdr6rpz85y455sfmz6vyf-source/mach_nix/generate.py", line 79, in handle_resolution_impossible
    f" - parent: {ri.parent.name}{ri.parent.extras if ri.parent.extras else None}:{ri.parent.version}"
AttributeError: 'Candidate' object has no attribute 'version'
AttributeError: 'Candidate' object has no attribute 'version'
builder for '/nix/store/5ssy5glf78kh973w5bz5ps82q2kri50n-mach_nix_file.drv' failed with exit code 1
while evaluating the attribute 'passAsFile' of the derivation 'python3-3.8.5-env' at /nix/store/pz3w468p7ig3zj8fy5xsbrjn95dwiiwj-nixpkgs/pkgs/build-support/trivial-builders.nix:7:7:
while evaluating 'callPackageWith' at /nix/store/pz3w468p7ig3zj8fy5xsbrjn95dwiiwj-nixpkgs/lib/customisation.nix:117:35, called from /nix/store/pz3w468p7ig3zj8fy5xsbrjn95dwiiwj-nixpkgs/pkgs/development/interpreters/python/default.nix:20:24:
while evaluating 'makeOverridable' at /nix/store/pz3w468p7ig3zj8fy5xsbrjn95dwiiwj-nixpkgs/lib/customisation.nix:67:24, called from /nix/store/pz3w468p7ig3zj8fy5xsbrjn95dwiiwj-nixpkgs/lib/customisation.nix:121:8:
while evaluating anonymous function at /nix/store/pz3w468p7ig3zj8fy5xsbrjn95dwiiwj-nixpkgs/pkgs/top-level/python-packages.nix:9:1, called from /nix/store/pz3w468p7ig3zj8fy5xsbrjn95dwiiwj-nixpkgs/lib/customisation.nix:69:16:
while evaluating 'fix'' at /nix/store/pz3w468p7ig3zj8fy5xsbrjn95dwiiwj-nixpkgs/lib/fixed-points.nix:25:10, called from /nix/store/pz3w468p7ig3zj8fy5xsbrjn95dwiiwj-nixpkgs/pkgs/top-level/python-packages.nix:7738:4:
while evaluating 'extends' at /nix/store/pz3w468p7ig3zj8fy5xsbrjn95dwiiwj-nixpkgs/lib/fixed-points.nix:69:24, called from /nix/store/pz3w468p7ig3zj8fy5xsbrjn95dwiiwj-nixpkgs/lib/fixed-points.nix:25:21:
while evaluating 'composeExtensions' at /nix/store/pz3w468p7ig3zj8fy5xsbrjn95dwiiwj-nixpkgs/lib/fixed-points.nix:75:17, called from /nix/store/pz3w468p7ig3zj8fy5xsbrjn95dwiiwj-nixpkgs/lib/fixed-points.nix:69:67:
while evaluating the attribute 'overrides' at /nix/store/ch8zsbs2vlfqdr6rpz85y455sfmz6vyf-source/default.nix:37:7:

error: --- Error --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-shell
build of '/nix/store/5ssy5glf78kh973w5bz5ps82q2kri50n-mach_nix_file.drv' failed
DavHau commented 4 years ago

Thanks for that! You where hitting a bug in printing information about the resolution error. It's fixed now in the dev branch.

Your expression calls mach-nix.buildPythonPackage to build deeplabcut from github but the requirements you put in are not from deeplabcat. Therefore I assume you just copied the expression head from somewhere else and you actually want to build a python environment, not a python package.

I fixed that and tried to run your expression through mach-nix

The error messages I was getting revealed that your requirements contain a lot of conflicts. I guess you used some exporter to retrieve them. (Pip might accept this since pip just ignores conflicts)

I fixed those conflicts one by one, to see if we're hitting any other bugs and found that mach-nix cannot deal witch python packages which have a dot in their name. I fixed this also on the dev branch. Now your environment is working

Fixing the conflicts requires the following changes yo your requirements:

<     dask==2.22.0
---
>     dask[complete]==2.22.0
78c78
<     llvmlite==0.34.0rc1
---
> 
103c103
<     osmpythontools==0.2.8
---
>     osmpythontools
127c127
<     pyct==0.4.6
---
>     pyct[cmd]==0.4.6
149c149
<     rasterio==1.1.5
---
>     rasterio[s3]==1.1.5

The resulting working nix expression is: (also see the provider config at the bottom)

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "dev";
    rev = "371c0ebf7d6629c6893bc140be541452b2bd3129";
  });
in mach-nix.mkPython {
  requirements = ''
    affine==2.3.0
    appdirs==1.4.4
    #appnope==0.1.0; python_version >= "3.3" and sys_platform == "darwin" or platform_system == "Darwin"
    arcgis==1.8.2
    area==1.1.1
    argon2-cffi==20.1.0
    #atomicwrites==1.4.0; sys_platform == "win32"
    attrs==19.3.0
    autoflake==1.3.1
    backcall==0.2.0
    beautifulsoup4==4.9.1
    black==19.10b0
    bleach==3.1.5
    bokeh==2.1.1
    certifi==2020.6.20
    cffi==1.14.1
    chardet==3.0.4
    click==7.1.2
    click-plugins==1.1.1
    cligj==0.5.0
    #colorama==0.4.3; python_version >= "3.3" and sys_platform == "win32" or sys_platform == "win32"
    colorcet==2.0.2
    cryptography==3.0
    cycler==0.10.0
    dask[complete]==2.22.0
    datashader==0.10.0
    datashape==0.5.2
    decorator==4.4.2
    defusedxml==0.6.0
    descartes==1.1.0
    entrypoints==0.3
    feather-format==0.4.1
    fiona==1.8.13.post1
    flake8==3.8.3
    flufl.enum==4.1.1
    geoalchemy2==0.8.4
    geofeather==0.3.0
    geographiclib==1.50
    geojson==2.5.0
    geojsonio==0.0.3
    geolinks==0.2.0
    geopandas==0.8.1
    geopy==2.0.0
    github3.py==1.3.0
    googledrivedownloader==0.4
    greedy==0.1.1
    idna==2.10
    imageio==2.9.0
    importlib-metadata==1.7.0
    ipykernel==5.3.4
    ipython==7.17.0
    ipython-genutils==0.2.0
    ipywidgets==7.5.1
    jedi==0.17.2
    jeepney==0.4.3
    jinja2==2.11.2
    joblib==0.16.0
    json5==0.9.5
    jsonschema==3.2.0
    jupyter-client==6.1.6
    jupyter-core==4.6.3
    jupyterlab==2.2.4
    jupyterlab-server==1.2.0
    jwcrypto==0.7
    keyring==21.3.0
    kiwisolver==1.2.0
    lerc==0.1.0
    libpysal==4.3.0
    lidar==0.5.0

    lxml==4.5.2
    markupsafe==1.1.1
    matplotlib==3.3.0
    mccabe==0.6.1
    mercantile==1.1.5
    missingno==0.4.2
    mistune==0.8.4
    momepy==0.3.0
    more-itertools==8.4.0
    multipledispatch==0.6.0
    munch==2.5.0
    mypy==0.782
    mypy-extensions==0.4.3
    nbconvert==5.6.1
    nbformat==5.0.7
    networkx==2.4
    notebook==6.1.1
    ntlm-auth==1.5.0
    numba==0.50.1
    numpy==1.19.1
    oauthlib==3.1.0
    osm-diff-tool==1.1.0
    osmapi==1.2.2
    osmnet==0.1.6
    osmpythontools
    overpass==0.7
    overpy2==0.4.2
    owslib==0.20.0
    packaging==20.4
    pandas==1.1.0
    pandocfilters==1.4.2
    param==1.9.3
    parso==0.7.1
    pathlib==1.0.1
    pathspec==0.8.0
    pexpect==4.8.0
    pickleshare==0.7.5
    pillow==7.2.0
    pluggy==0.13.1
    pooch==1.1.1
    prometheus-client==0.8.0
    prompt-toolkit==3.0.5
    ptyprocess==0.6.0
    py==1.9.0
    pyarrow==1.0.0
    pycodestyle==2.6.0
    #pycparser==2.20; platform_system == "Windows" and python_version != "3.3" or platform_system == "Windows" and python_version != "3.3" and sys_platform == "linux"
    pycsw==2.4.2
    pyct[cmd]==0.4.6
    pydal==20200714.1
    pyflakes==2.2.0
    pygeos==0.7.1
    pygis==0.1.3
    pygments==2.6.1
    #pykerberos==1.2.1; platform_system == "Windows" and sys_platform != "win32"
    pyparsing==2.4.7
    pyproj==2.6.1.post1
    pyrsistent==0.16.0
    pyshp==2.1.0
    pysimplegui==4.28.0
    pytest==5.4.3
    python-dateutil==2.8.1
    pytz==2020.1
    pywavelets==1.1.1
    #pywin32==228; sys_platform == "win32" or platform_system == "Windows"
    #pywin32-ctypes==0.2.0; sys_platform == "win32"
    #pywinpty==0.5.7; os_name == "nt"
    pywps==4.2.7
    pyyaml==5.3.1
    pyzmq==19.0.2
    rasterio[s3]==1.1.5
    rasterstats==0.15.0
    regex==2020.7.14
    requests==2.24.0
    #requests-kerberos==0.12.0; platform_system == "Windows"
    requests-ntlm==1.1.0
    requests-oauthlib==1.3.0
    requests-toolbelt==0.9.1
    richdem==0.3.4
    rio-cogeo==1.1.10
    rsgis==0.0.3
    scikit-image==0.17.2
    scikit-learn==0.23.2
    scipy==1.5.2
    seaborn==0.10.1
    secretstorage==3.1.2
    send2trash==1.5.0
    shapely==1.7.0
    simplejson==3.17.2
    six==1.15.0
    snuggs==1.4.7
    soupsieve==1.9.6
    spatial==0.2.0
    sqlalchemy==1.3.18
    supermercado==0.1.1
    terminado==0.8.3
    testpath==0.4.4
    threadpoolctl==2.1.0
    tifffile==2020.7.24
    toml==0.10.1
    toolz==0.10.0
    tornado==6.0.4
    tqdm==4.48.2
    traitlets==4.3.3
    typed-ast==1.4.1
    typing-extensions==3.7.4.2
    ujson==3.1.0
    uritemplate==3.0.1
    urllib3==1.25.10
    verde==1.5.0
    wcwidth==0.2.5
    webencodings==0.5.1
    werkzeug==1.0.1
    widgetsnbextension==3.5.1
    #winkerberos==0.7.0; platform_system == "Windows" and sys_platform == "win32" or platform_system == "Windows"
    xarray==0.16.0
    xarray-spatial==0.0.8
    xmltodict==0.12.0
    zipp==3.1.0
  '';

  providers = {
    # Change the provider for osmpythontools to "nixpkgs", because the inherited patches from nixpkgs won't apply. 
    osmpythontools = "nixpkgs";
  };
}
InLaw commented 4 years ago

eg. rasterio==1.1.5 > rasterio[s3]==1.1.5 and to choose the best provider for osmpythontools

it shows:

Multiple nixkgs attributes found for tornado-6.0.4: ['tornado_4', 'tornado']
Picking 'tornado' as base attribute name.

InLaw commented 4 years ago

why is tornado installed as "tornado - 6.0.4 - sdist" if

providers = {

    _default = "nixpkgs,wheel,sdist";
}
DavHau commented 4 years ago

How could I debug the changes by myself? eg. rasterio==1.1.5 > rasterio[s3]==1.1.5 and to choose the best provider for osmpythontools

You can either install the rasterio package with s3 or without s3 but you cannot have both variants at the same time, therefore the resolver raises a conflict. Therefore if your top level requirements contain rasterio==1.1.5 and a conflict appears between rasterio and rasterio[s3], you know that one of your subdependencies must have declared rasterio[s3]. This can either be fixed by adapting your top level dependencies or removing the sub dependency which requires rasterio[s3].

and to choose the best provider for osmpythontools

My train of thought here:

  1. The build of osmpythontools failed.
  2. The build log shows there have been problems with patches.
  3. The provider was sdist ( -> means no wheel available)
  4. Packages from sdist provider inherit patches from nixpkgs. In this case they probably don't fit with the current version of osmpythontools.

Now I can try one of two things:

  1. Remove the patches by using overrides_pre to set patches = []
  2. Change the provider. Since wheel is not available, the only other option is nixpkgs.

Multiple nixkgs attributes found for tornado-6.0.4: ['tornado_4', 'tornado'] Picking 'tornado' as base attribute name. what does it mean?

For different reasons, like inheriting patches or excluding the chance of collisions, mach-nix must know about all candidates for a package in nixpkgs. Also if you're not using the nixpkgs provider. It will always try to override the candidate which seems most similar (closest version) than the one which is built. If it cannot decide, because the versions are too far apart, it shows this message and just picks one of both. Maybe it's confusing and I should hide it since it doesn't really matter in most cases.

there should be a "python3.7-tornado-6.0.4 | python37Packages.tornado"

Tornado only releases wheels for windows

is it possible to activate a cache for wheel packages?

Why would you want to do that? Wheel packages are very quick to install. If they contain extension modules, they are already in binary format and just need to be patched. There wouldn't be much gain in caching them. Caching sdist packages would be more of a benefit.

InLaw commented 4 years ago

From my point of view, it is always good to get those information (so that you are aware of what is happening)