API to parse requirements.txt for setup.py

[techtonik]

requirements.txt could a convenient single declarative point for specifying dependencies if it could be pasted into setup.py install_requires as-is. But requirements.txt contains comments and references to other requirements files, which seem to be incompatible with install_requires (reference needed).

It would be nice to have official API that parses requirements.txt file into a list suitable for install_requires section.

https://stackoverflow.com/questions/14399534/reference-requirements-txt-for-the-install-requires-kwarg-in-setuptools-setup-py

[benoit-pierre]

There's also no support for environment markers in install_requires requirements (e.g. python-xlib==0.19; "linux" in sys_platform will not work), something that is handled by extras_require.

Note that install_requires can just be a multi-lines string, so if you stick to a subset of pip's requirements format that is supported by setuptools, you should be able to use something as simple as:

with open('requirements.txt') as fp:
    install_requires = fp.read()

setup(install_requires=install_requires, ...)

in your setup.py.

Also looks like a duplicate of #1074.

[techtonik]

Note that install_requires can just be a multi-lines string

Nice. Save me a lot of time. If only I could detect when requirements.txt becomes incompatible without waiting for setup.py to fail.

so if you stick to a subset of pip's requirements format that is supported by setuptools

Are those differences documented somewhere?

[benoit-pierre]

Well, at the very least, setuptools should error out if environment markers are used:

 setuptools/dist.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git i/setuptools/dist.py w/setuptools/dist.py
index 6b97ed33..eb9e4442 100644
--- i/setuptools/dist.py
+++ w/setuptools/dist.py
@@ -153,7 +153,11 @@ def assert_bool(dist, attr, value):
 def check_requirements(dist, attr, value):
     """Verify that install_requires is a valid requirements list"""
     try:
-        list(pkg_resources.parse_requirements(value))
+        reqs = []
+        for r in pkg_resources.parse_requirements(value):
+            if r.marker:
+                raise ValueError("environment markers are not supported, in '%s'" % r)
+            reqs.append(r)
     except (TypeError, ValueError) as error:
         tmpl = (
             "{attr!r} must be a string or list of strings "

And the documentation should be fixed, as both direct install with setup.py and from the generated wheel may fail.

[benoit-pierre]

After inspection, environment markers do work when installing from source, but not with the generated wheels, see #1081.

[jaraco]

Agreed this is a duplicate of #1074.

[tuukkamustonen]

Might be a stupid question, but does pip's requirements.txt format even support named extras_requires dependencies? E.g. parsing/converting following from a requirements.txt file:

extras_requires={
    'extra_scope': 'somepackage >= 1.0.0'
}

From like:

somepackage >= 1.0.0; [extra_scope]

?

[techtonik]

1074 needs a better title.

Also, there are two scenarios in parsing requirements.txt which add the complexity into resolution of this issue:

1. parsing requirements.txt just to get structured data about all package dependencies (including optional and platform specific)
2. parsing requirements.txt to match requirements with specific current system

2 is actually two step process, which includes step 1. But because 2 is more often needed than 1, it is mixed up together and makes API hard.

[techtonik]

Might be a stupid question, but does pip's requirements.txt format even support named extras_requires dependencies?

@tuukkamustonen https://stackoverflow.com/a/43090648/239247 - is that it?

[tuukkamustonen]

https://stackoverflow.com/a/43090648/239247 - is that it?

@techtonik That is for depending on named extras in other packages. I'm after being able to declare them in mine, so to say something like pip install -r requirements.txt [extra1, extra2] (like in pip install .[extra1, extra2].

So some format that I can parse from requirements.txt into setup.pys:

extras_requires={
    'extra_scope': 'somepackage >= 1.0.0'
}

[techtonik]

@tuukkamustonen so, you want a mechanism to specify different sets of optional dependencies? I believe it is done with tags specified in square brackets.

somepackage >= 1.0.0 [extra_scope]

https://www.python.org/dev/peps/pep-0508/#extras https://stackoverflow.com/questions/3664478/optional-dependencies-in-a-pip-requirements-file

[tuukkamustonen]

@techtonik That syntax just gives a parsing error (in pip 9.0.1).

[techtonik]

@tuukkamustonen then it is a bug.

[jaraco]

@tuukkamustonen You use setup.py to declare your package's dependencies (and extras). You use requirements.txt to install any number of packages (and their dependencies).

So if you want to declare an extra scope for your package, you do that in setup.py:

extras_require={
  'extra_scope': [
    'somepackage >= 1.0.0',
  ],
}

Then, if you want to install your package with the extra scope, you can pip install .[extra_scope] or put .[extra_scope] in your requirements.txt or if you've uploaded your package to pypi, you can pip install your_package[extra_scope] or pip install your_package[extra_scope] >= 9.0.

Does that help?

[tuukkamustonen]

@jaraco Thanks for the write-up, but my question was if I can do that (extras_require declaration) in requirements.txt. So to define named extras in requirements.txt.

The reason I asked this in the context of this ticket is that if I wanted to parse named extras from requirements.txt into extras_require programmatically (even though everyone is recommending against it, but this ticket is about supporting that use case anyway), the API that this ticket suggests, should also support that. But that leads to a question if requirements.txt syntax even supports named extras.

Maybe an easier to understand use case would be this: Let's forget setup.py altogether and say that I have dev-requirements.txt which define requirements for being able to build and upload my project. In addition to that I could have test-requirements.txt and doc-requirements.txt for also being able to run tests and compile documentation.

Why would I split the dev requirements into several files? In CI (e.g. Jenkins), not every job needs all the dependencies, and maybe I want to optimize the time it takes to setup a virtualenv (let's say I'm spinning up a new Docker container for each build).

In order to avoid cluttering my repository with *-requirements.txt files, it would be fun to describe them all in one dev-requirements.txt, as in:

# Base
setuptools
twine >= 1.0.0

# Tests
pytest >= 2.0.0 [tests]

# Doc
sphinx [tests]

And then being able to run something like:

pip install -r dev-requirements.txt
pip install -r dev-requirements.txt [tests]  # yeah I don't think this is supported?
pip install -r dev-requirements.txt [doc]

I followed the syntax suggested by https://github.com/pypa/setuptools/issues/1080#issuecomment-342147121 here (but note that it gives a parsing error). According to @techtonik that's a bug. There may be some confusion here, so is it really a bug, or just feature that is not specified/supported?

[benoit-pierre]

requirements.txt does not support optional requirements sections. Extras are supported for asking for a requirement optional dependencies to be installed, not for declaring optional requirements themselves. For example, you can use:

pytest-runner [testing]

To include all the testing dependencies required by pytest-runner, but that's not a way to declare an optional testing requirement. The format used by requirements.txt does not support that (unlike the format used by requires.txt as part of egg-info).

[tuukkamustonen]

So that's just not supported then. Ok. Thanks for clarifying things up.

[pganssle]

Given that this is a duplicate of #1074 and that #1074 is closed as wontfix, are we also closing this as wontfix?

[techtonik]

@pganssle there is no proposal in #1074, so nothing to fix, and here I made sure that there is a proposal to make a way for programmatic access to requirements.txt files.

[mattvonrocketstein]

requirements.txt supports this, setup.py does not: git+https://github.com/user/repo.git@master#egg=loggable.

setup.py gives an error like this: error in setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+https:/'"

stuff like this this is still a valid question and a confusing issue years and years later.

this api should exist and maybe parse requirements.txt to return a dictionary of {"dependency_links": [...], "install_requires": [....]} which is suitable to use like this: setup(<other_setup_keywords>, **parse_requirements('requirements.txt'))

[sam-writer]

Note that install_requires can just be a multi-lines string, so if you stick to a subset of pip's requirements format that is supported by setuptools, you should be able to use something as simple as:

with open('requirements.txt') as fp:
    install_requires = fp.read()

setup(install_requires=install_requires, ...)

in your setup.py.

This can cause problems with hyphenated package names. It is fairly common for PyPi packages to be listed with a hyphen in their name, but for all other references to them to have underscores. An example package is pytorch-transformers, where you pip install pytorch-transformers, but when using, you import pytorch_transformers.

Because of this, you need to replace - with _. I tried to parse things myself, but after realizing that a line in requirements.txt could be a URL, or something like mymodule>1.5,<1.6, I decided that the parser I was writing must already exist... and it does, requirements-parser. So, using that, I've got:

import requirements

install_requires = []

with open('requirements.txt') as fd:
    for req in requirements.parse(fd):
        if req.name:
            name = req.name.replace("-", "_")
            full_line = name + "".join(["".join(list(spec)) for spec in req.specs])
            install_requires.append(full_line)
        else:
            # a GitHub URL doesn't have a name
            # I am not sure what to do with this, actually, right now I just ignore it...
setup(install_requires=install_requires, ...)

It's not pretty, and it adds a dependency, though.

[techtonik]

Judging from https://github.com/davidfischer/requirements-parser/issues/45 the author probably would not object if requirements-parser would be merged into pypa.