Open techtonik opened 7 years ago
There's also no support for environment markers in install_requires
requirements (e.g. python-xlib==0.19; "linux" in sys_platform
will not work), something that is handled by extras_require
.
Note that install_requires
can just be a multi-lines string, so if you stick to a subset of pip's requirements format that is supported by setuptools, you should be able to use something as simple as:
with open('requirements.txt') as fp:
install_requires = fp.read()
setup(install_requires=install_requires, ...)
in your setup.py
.
Also looks like a duplicate of #1074.
Note that install_requires can just be a multi-lines string
Nice. Save me a lot of time. If only I could detect when requirements.txt
becomes incompatible without waiting for setup.py
to fail.
so if you stick to a subset of pip's requirements format that is supported by setuptools
Are those differences documented somewhere?
Well, at the very least, setuptools should error out if environment markers are used:
setuptools/dist.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git i/setuptools/dist.py w/setuptools/dist.py
index 6b97ed33..eb9e4442 100644
--- i/setuptools/dist.py
+++ w/setuptools/dist.py
@@ -153,7 +153,11 @@ def assert_bool(dist, attr, value):
def check_requirements(dist, attr, value):
"""Verify that install_requires is a valid requirements list"""
try:
- list(pkg_resources.parse_requirements(value))
+ reqs = []
+ for r in pkg_resources.parse_requirements(value):
+ if r.marker:
+ raise ValueError("environment markers are not supported, in '%s'" % r)
+ reqs.append(r)
except (TypeError, ValueError) as error:
tmpl = (
"{attr!r} must be a string or list of strings "
And the documentation should be fixed, as both direct install with setup.py
and from the generated wheel may fail.
After inspection, environment markers do work when installing from source, but not with the generated wheels, see #1081.
Agreed this is a duplicate of #1074.
Might be a stupid question, but does pip's requirements.txt
format even support named extras_requires
dependencies? E.g. parsing/converting following from a requirements.txt file:
extras_requires={
'extra_scope': 'somepackage >= 1.0.0'
}
From like:
somepackage >= 1.0.0; [extra_scope]
?
Also, there are two scenarios in parsing requirements.txt
which add the complexity into resolution of this issue:
requirements.txt
just to get structured data about all package dependencies (including optional and platform specific)requirements.txt
to match requirements with specific current system2 is actually two step process, which includes step 1. But because 2 is more often needed than 1, it is mixed up together and makes API hard.
Might be a stupid question, but does pip's requirements.txt format even support named extras_requires dependencies?
@tuukkamustonen https://stackoverflow.com/a/43090648/239247 - is that it?
https://stackoverflow.com/a/43090648/239247 - is that it?
@techtonik That is for depending on named extras in other packages. I'm after being able to declare them in mine, so to say something like pip install -r requirements.txt [extra1, extra2]
(like in pip install .[extra1, extra2]
.
So some format that I can parse from requirements.txt
into setup.py
s:
extras_requires={
'extra_scope': 'somepackage >= 1.0.0'
}
@tuukkamustonen so, you want a mechanism to specify different sets of optional dependencies? I believe it is done with tags specified in square brackets.
somepackage >= 1.0.0 [extra_scope]
https://www.python.org/dev/peps/pep-0508/#extras https://stackoverflow.com/questions/3664478/optional-dependencies-in-a-pip-requirements-file
@techtonik That syntax just gives a parsing error (in pip 9.0.1).
@tuukkamustonen then it is a bug.
@tuukkamustonen You use setup.py to declare your package's dependencies (and extras). You use requirements.txt to install any number of packages (and their dependencies).
So if you want to declare an extra scope for your package, you do that in setup.py:
extras_require={
'extra_scope': [
'somepackage >= 1.0.0',
],
}
Then, if you want to install your package with the extra scope, you can pip install .[extra_scope]
or put .[extra_scope]
in your requirements.txt or if you've uploaded your package to pypi, you can pip install your_package[extra_scope]
or pip install your_package[extra_scope] >= 9.0
.
Does that help?
@jaraco Thanks for the write-up, but my question was if I can do that (extras_require
declaration) in requirements.txt
. So to define named extras in requirements.txt
.
The reason I asked this in the context of this ticket is that if I wanted to parse named extras from requirements.txt
into extras_require
programmatically (even though everyone is recommending against it, but this ticket is about supporting that use case anyway), the API that this ticket suggests, should also support that. But that leads to a question if requirements.txt
syntax even supports named extras.
Maybe an easier to understand use case would be this: Let's forget setup.py
altogether and say that I have dev-requirements.txt
which define requirements for being able to build and upload my project. In addition to that I could have test-requirements.txt
and doc-requirements.txt
for also being able to run tests and compile documentation.
Why would I split the dev requirements into several files? In CI (e.g. Jenkins), not every job needs all the dependencies, and maybe I want to optimize the time it takes to setup a virtualenv (let's say I'm spinning up a new Docker container for each build).
In order to avoid cluttering my repository with *-requirements.txt
files, it would be fun to describe them all in one dev-requirements.txt
, as in:
# Base
setuptools
twine >= 1.0.0
# Tests
pytest >= 2.0.0 [tests]
# Doc
sphinx [tests]
And then being able to run something like:
pip install -r dev-requirements.txt
pip install -r dev-requirements.txt [tests] # yeah I don't think this is supported?
pip install -r dev-requirements.txt [doc]
I followed the syntax suggested by https://github.com/pypa/setuptools/issues/1080#issuecomment-342147121 here (but note that it gives a parsing error). According to @techtonik that's a bug. There may be some confusion here, so is it really a bug, or just feature that is not specified/supported?
requirements.txt
does not support optional requirements sections. Extras are supported for asking for a requirement optional dependencies to be installed, not for declaring optional requirements themselves. For example, you can use:
pytest-runner [testing]
To include all the testing dependencies required by pytest-runner, but that's not a way to declare an optional testing
requirement. The format used by requirements.txt
does not support that (unlike the format used by requires.txt
as part of egg-info).
So that's just not supported then. Ok. Thanks for clarifying things up.
Given that this is a duplicate of #1074 and that #1074 is closed as wontfix, are we also closing this as wontfix?
@pganssle there is no proposal in #1074, so nothing to fix, and here I made sure that there is a proposal to make a way for programmatic access to requirements.txt
files.
requirements.txt supports this, setup.py does not: git+https://github.com/user/repo.git@master#egg=loggable
.
setup.py gives an error like this: error in setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'+https:/'"
stuff like this this is still a valid question and a confusing issue years and years later.
this api should exist and maybe parse requirements.txt to return a dictionary of {"dependency_links": [...], "install_requires": [....]}
which is suitable to use like this: setup(<other_setup_keywords>, **parse_requirements('requirements.txt'))
Note that
install_requires
can just be a multi-lines string, so if you stick to a subset of pip's requirements format that is supported by setuptools, you should be able to use something as simple as:with open('requirements.txt') as fp: install_requires = fp.read() setup(install_requires=install_requires, ...)
in your
setup.py
.
This can cause problems with hyphenated package names. It is fairly common for PyPi packages to be listed with a hyphen in their name, but for all other references to them to have underscores. An example package is pytorch-transformers, where you pip install pytorch-transformers
, but when using, you import pytorch_transformers
.
Because of this, you need to replace -
with _
. I tried to parse things myself, but after realizing that a line in requirements.txt
could be a URL, or something like mymodule>1.5,<1.6
, I decided that the parser I was writing must already exist... and it does, requirements-parser. So, using that, I've got:
import requirements
install_requires = []
with open('requirements.txt') as fd:
for req in requirements.parse(fd):
if req.name:
name = req.name.replace("-", "_")
full_line = name + "".join(["".join(list(spec)) for spec in req.specs])
install_requires.append(full_line)
else:
# a GitHub URL doesn't have a name
# I am not sure what to do with this, actually, right now I just ignore it...
setup(install_requires=install_requires, ...)
It's not pretty, and it adds a dependency, though.
Judging from https://github.com/davidfischer/requirements-parser/issues/45 the author probably would not object if requirements-parser
would be merged into pypa
.
requirements.txt
could a convenient single declarative point for specifying dependencies if it could be pasted intosetup.py
install_requires
as-is. Butrequirements.txt
contains comments and references to other requirements files, which seem to be incompatible withinstall_requires
(reference needed).It would be nice to have official API that parses
requirements.txt
file into a list suitable forinstall_requires
section.https://stackoverflow.com/questions/14399534/reference-requirements-txt-for-the-install-requires-kwarg-in-setuptools-setup-py