search

conda / conda-lock

Lightweight lockfile for conda environments
https://conda.github.io/conda-lock/
Other
488 stars 103 forks source link

Support for full pip requirements.txt syntax in environment.yml? #171

Open lesteve opened 2 years ago

lesteve commented 2 years ago

This may be stretching conda-lock a bit, but would there be a chance to support full pip requirements.txt syntax in environment.yml like the following one? This is supported by conda and mamba, see https://github.com/conda/conda/pull/3969 for conda support:

# /tmp/environment.yml
channels:
  - conda-forge

dependencies:
  - pip
  - pip:
    - --pre
    - --extra-index https://pypi.anaconda.org/scipy-wheels-nightly/simple
    - numpy
    - https://github.com/joblib/joblib/archive/master.zip

The use case we have in scikit-learn:

Right now I get an error when trying to lock the environment because --pre does not look like a "standard" requirement e.g. something like package>version.

The full output with the error:

❯ conda-lock lock --mamba -p linux-64 -f /tmp/environment.yml 
Traceback (most recent call last):
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 102, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 1654, in parseString
    raise exc
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 1644, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 3417, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 3739, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 3400, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 1406, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 2711, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lesteve/miniconda3/envs/conda-lock/bin/conda-lock", line 33, in <module>
    sys.exit(load_entry_point('conda-lock', 'console_scripts', 'conda-lock')())
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 1166, in lock
    lock_func(
  File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 936, in run_lock
    make_lock_files(
  File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 335, in make_lock_files
    lock_spec = make_lock_spec(
  File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 241, in make_lock_spec
    lock_specs = parse_source_files(
  File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 770, in parse_source_files
    parse_environment_file(src_file, pip_support=PIP_SUPPORT)
  File "/home/lesteve/dev/conda-lock/conda_lock/src_parser/environment_yaml.py", line 97, in parse_environment_file
    parse_python_requirement(
  File "/home/lesteve/dev/conda-lock/conda_lock/src_parser/pyproject_toml.py", line 239, in parse_python_requirement
    parsed_req = Requirement.parse(requirement_specifier)
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3134, in parse
    req, = parse_requirements(s)
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3089, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 104, in __init__
    raise InvalidRequirement(
pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at "'--pre'": Expected W:(abcd...)
mariusvniekerk commented 2 years ago

So part of that is possible and part of it is not entirely,

Since the pip solver being used is the poetry solver the configuration for allow-prereleases is not a global setting but a local one per dependency as defined over here https://github.com/python-poetry/poetry-core/blob/ffc22b22a2e9d075ba93fea1348729bf9af2c3fd/src/poetry/core/packages/dependency.py#L39 I assume in this case you probably want just a prerelease of just numpy, and not everything else.

The pypi repositories being used are defined in https://github.com/conda-incubator/conda-lock/blob/9a1132d703dd34a5524becba15b976769781d52d/conda_lock/pypi_solver.py#L200-L201 , there is already a feature request to support those a bit better in #135 so adding in that should potentially be feasible.

mariusvniekerk commented 2 years ago

The harder part is which pip install arguments to support generically as there are quite a few of them and the mapping of them to poetry flags isn't necessarily trivial

lesteve commented 2 years ago

I assume in this case you probably want just a prerelease of just numpy, and not everything else.

Yep indeed in practice we would want prerelease versions only for a few packages e.g. numpy, scipy, pandas so it would be fine to do it on a per-package basis.

there is already a feature request to support those a bit better in https://github.com/conda-incubator/conda-lock/issues/135 so adding in that should potentially be feasible.

OK nice.

There was actually a third hidden feature request in my original post: having dependencies like https://github.com/joblib/joblib/archive/master.zip (and I assume git+https:// as well). Right now I am getting a parsing error:

pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at "'://githu'": Expected stringEnd
Full error ``` conda-lock ❯ conda-lock lock --mamba -p linux-64 -f /tmp/test-environment.yml --lockfile /tmp/test-conda-lock.yml Traceback (most recent call last): File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 102, in __init__ req = REQUIREMENT.parseString(requirement_string) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 1654, in parseString raise exc File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 1644, in parseString loc, tokens = self._parse( instring, 0 ) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache loc,tokens = self.parseImpl( instring, preloc, doActions ) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 3417, in parseImpl loc, exprtokens = e._parse( instring, loc, doActions ) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 1406, in _parseNoCache loc,tokens = self.parseImpl( instring, preloc, doActions ) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/pyparsing.py", line 3205, in parseImpl raise ParseException(instring, loc, self.errmsg, self) pkg_resources._vendor.pyparsing.ParseException: Expected stringEnd (at char 5), (line:1, col:6) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/lesteve/miniconda3/envs/conda-lock/bin/conda-lock", line 33, in sys.exit(load_entry_point('conda-lock', 'console_scripts', 'conda-lock')()) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 1166, in lock lock_func( File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 936, in run_lock make_lock_files( File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 335, in make_lock_files lock_spec = make_lock_spec( File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 241, in make_lock_spec lock_specs = parse_source_files( File "/home/lesteve/dev/conda-lock/conda_lock/conda_lock.py", line 770, in parse_source_files parse_environment_file(src_file, pip_support=PIP_SUPPORT) File "/home/lesteve/dev/conda-lock/conda_lock/src_parser/environment_yaml.py", line 97, in parse_environment_file parse_python_requirement( File "/home/lesteve/dev/conda-lock/conda_lock/src_parser/pyproject_toml.py", line 239, in parse_python_requirement parsed_req = Requirement.parse(requirement_specifier) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3134, in parse req, = parse_requirements(s) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3089, in __init__ super(Requirement, self).__init__(requirement_string) File "/home/lesteve/miniconda3/envs/conda-lock/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 104, in __init__ raise InvalidRequirement( pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at "'://githu'": Expected stringEnd ```
rcthomas commented 1 year ago

Maybe this should be a separate issue, but I think it's related. I was wondering how to get conda-lock to install pip dependencies a la pip install --no-cache-dir ... instead of manually deleting the cache before running conda-lock install. Instead I think my question should be restated to is there a way to get conda-lock to get poetry to clear the cache before building those dependencies?

maresb commented 1 year ago

@rcthomas, is this motivated by reducing disk space for Docker or other space-constrained environments?

Unfortunately I don't think there's a good way to this automatically with the way things are currently structured. Manually deleting the cache seems to me like the way to go.

rcthomas commented 1 year ago

@maresb no, not disk space, I just sometimes want to make sure that I'm not using the pip cache directory when building packages from PyPI.

The reason I followed up on this thread instead of opening a new one is that I'm actually not looking for a way to do this automatically, I was wondering if I could tell conda-lock to pass arguments like OP asked about, but to prevent using the pip cache (as if one were passing --no-cache-dir to pip install). Initially I thought "if there's just a way to pass arguments to pip" that would work but I do see why it's maybe not as easy as that.

tbenst commented 8 months ago

A simpler but related usecase, would be nice if requirements.txt was supported in environment.yml

dependencies:
  - python=3.11
  - pip
  - pip:
    - -r requirements.txt
    - conda-lock
maresb commented 8 months ago

Ya, I see this thread as good motivation to eliminate conda-lock's use of Poetry, replacing it with a generic solver.

ollie-bell commented 3 months ago

Since the pip solver being used is the poetry solver the configuration for allow-prereleases is not a global setting but a local one per dependency as defined over here https://github.com/python-poetry/poetry-core/blob/ffc22b22a2e9d075ba93fea1348729bf9af2c3fd/src/poetry/core/packages/dependency.py#L39 I assume in this case you probably want just a prerelease of just numpy, and not everything else.

How does one get the poetry dependency solver in conda-lock to set allow_prereleases to True when finding the appropriate package version? Is this something that can be done with some special syntax in the environment.yaml file?

maresb commented 3 months ago

I don't think we currently have any way to activate allow_prereleases=True. I think you'd have to patch the source. If you add an option to pyproject.toml and a test I wouldn't see any problem with merging it in a PR.

ollie-bell commented 3 months ago

Ah ok. That's a shame because pip's default behaviour is to find pre-releases when it can't find a stable release (which is the issue in my case... installing the conda environment works fine but exporting it using conda-lock fails). I'll look into it but not familiar with poetry at all!