Support for setup.py

[LegoStormtroopr]

I need requirements in setup.py in the install_requires config setting, but these aren't picked up in setup.py.

I know its probably difficult, but do you have any plans on support for this?

[ntravis]

From https://packaging.python.org/requirements/

It is not considered best practice to use install_requires to pin dependencies to specific versions, or to specify sub-dependencies (i.e. dependencies of your dependencies). This is overly-restrictive, and prevents the user from gaining the benefit of dependency upgrades.

The best practice they indicate above that statement is to indicate any known lower bounds, and if you know the package you are requiring follows semantic versioning (or some other, lower versioning) and that the next major version should include breaking changes, you may also indicate an upper bound.

Can you post your setup.py and why you'd want this to be upgraded via pyup instead of using something like Travis to test builds with different versions of dependencies or something along those lines?

[LegoStormtroopr]

Thanks for linking that. I do think I was wrong in asking for support around this. I'm still figuring out how setup.py and requirements.txt interact with each other.

But the two important things are:

1. Even if I was wrong in asking, I was first to be wrong!
2. If people search for setup.py in the issues this will come up and they won't ask.

So what would be the recommended approach? Build with travis-ci using setup.py to manage installations with a matching requirements.txt to check version dependencies?

[ntravis]

It probably depends on what your goal. If you want to check your project against different versions of its dependencies, you should be able to instruct Travis to install different versions of your direct dependencies and run tests. https://docs.travis-ci.com/user/languages/python/#Testing-Against-Multiple-Versions-of-Dependencies-(e.g.-Django-or-Flask)

the "env" command could be used to specific several versions of your sub-dependencies if you are worried about testing them, otherwise it'll pull the latest version available if you are specifying >= or just the name of the requirement. Keep in mind that will only be tested when your code is pushed or triggered, not on their changes. If you want to get notifications about those sorts of changes, you could keep a requirements file to ensure that you are notified of their updates and will see the effect on your project. Is that what you're after?

[jayfk]

Nevertheless, I think supporting setup.py files in read-only mode would be a great feature.

This way libraries could use pyup.io's UI to get a quick glance about the dependencies.

[pawamoy]

Unpinned dependencies in setup.py's install_requires can lead to installing latest but insecure version of a package, so it could indeed be interesting to have it parsed and checked by safety, just for information (note that I do not recommend pinning deps in setup.py).

[jayfk]

I think there is a use case if you are using ranges like requests>=XY in setup.py. It would be great if the bot could notify you to update that range in case there's a fix available.

[keimlink]

We define a version range in setup.py:

setup(
    name='mydist',
    install_requires=[
        'Django>=1.8,<1.9',
    ]
)

Then we use a constraints file to specify the exact version:

django==1.8.17 # pyup: >=1.8,<1.9

constraints.pip is watched by pyup.io to update it as soon as a new Django version is available.

The distribution is then installed like this:

python3 -m pip install --constraint constraints.pip mydist

[DEKHTIARJonathan]

For those who don't want to wait, I have implement a small workaround to allow PyUP to process my setup.py file.

Please have a look to this repository: https://github.com/DEKHTIARJonathan/keras-datasets

Basically, you define txt files which contains your dependencies and you import them inside the setup.py file as a list. Done in 5 mins and 100% efficient ;)

Function used to import any requirements.txt file:

def req_file(filename):
    with open(filename) as f:
        content = f.readlines()
    # you may also want to remove whitespace characters like `\n` at the end of each line
    return [x.strip() for x in content] 

Which gives the following inside setup.py:

setup(
    name=__package_name__,
    install_requires=req_file("requirements_lib.txt"),
    extras_require={
        'dev': req_file("requirements_dev.txt"),
        'travis': req_file("requirements_travis.txt")
    }
)

Enjoy !

[edmorley]

Just to add to @DEKHTIARJonathan's workaround -- is it necessary to also set pin: False (in .pyup.yml) for the requirements file used to populate install_requires to prevent the version ranges (which are best practice for install_requires) from being converted to exact versions?

[DEKHTIARJonathan]

@edmorley Yeah I had to change it myself, forgot to precise it here. Sorry about that

[pombredanne]

@DEKHTIARJonathan re

For those who don't want to wait, I have implement a small workaround to allow PyUP to process my setup.py file.

You are not alone doing these kind of things, but this breaks so many concepts of packaging ... See @dstufft https://caremad.io/posts/2013/07/setup-vs-requirement/

Also see https://github.com/pyupio/dparse/issues/2 and https://github.com/nexB/scancode-toolkit/issues/253#issuecomment-310714287

@demosdemon any progress on your side? IMHO the only safe approach is AST parsing and not executing anything.

[bloodearnest]

Yeah, in my python libraries, documenting/enforcing minimum and maximum supported versions dependencies means I need install_requires (and extras_require) in setup.py.

I don't really like loading from a requirements.txt manually in setup.py, as that gets complex quickly (e.g. supporting environment markers and such).

Another factor is setuptools support for declarative metadata in setup.cfg (since 30.3.0, released Dec 2016):

https://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-setup-using-setup-cfg-files

I use this in some of my projects, be great if pyup could support it. Plus no arbitrary code execution!

As a variation on @DEKHTIARJonathan 's workaround, you can instead parse the dependencies from setup.cfg:

from setuptools.config import read_configuration
config = read_configuration('setup.cfg')
reqs = config['options']['install_requires']

[keimlink]

It's great that setuptools finally support declarative metadata. It has been a long way! 🎉

@bloodearnest Could you please give an example how you declare versions for extra dependencies in setup.cfg?

[bloodearnest]

Here's one example:

https://github.com/canonical-ols/talisker/blob/master/setup.cfg

[keimlink]

@bloodearnest Thanks for sharing the configuration. 👍 As far as I see you only define dependency configurations that use ranges, not exact versions. For example:

install_requires = 
    gunicorn>=19.5.0,<20.0
    Werkzeug>=0.11.5,<0.15
    statsd>=3.2.1,<4.0
    requests>=2.10.0,<3.0
    raven>=5.27.0,<7.0
    future>=0.15.2,<0.17
        ipaddress>=1.0.16,<2.0;python_version<"3.3"

I'm not sure, but I believe pyup can't update these. Right, @jayfk?

[jayfk]

Yes, that's right @keimlink. The only thing it could do is to pin them.

[bloodearnest]

Right.

This is a library, so it supports a range of dependency versions, as it needs to support different environments, as well as support upgrading dependencies independently. I suspect this would be the case in many pip installable libraries, rather than apps.

My personal preference would be for pyup to be able to limit the top end of the range, and leave the bottom end alone.

e.g. suppose my setup.py/cfg says: requests>=2.10.0,<=2.18.4, and theres a new 2.18.5 release, I'd love pyup to create PR updating my config to: requests>=2.10.0,<=2.18.5

That way I can use pyup to help explicitly and reliably support the latest releases of my dependencies, while making my own choices about what minimum version I support in my library (which depends on my users).

At the moment, I have to limit the top of the range by relying on semantic version guessing, and hope nothing breaks.

And regardless of range support, I think setup.cfg support would be useful, I expect more code will use this going forward.

Thanks for listening :)

On 1 March 2018 at 20:14, Jannis Gebauer notifications@github.com wrote:

Yes, that's right @keimlink https://github.com/keimlink. The only thing it could do is to pin them.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pyupio/pyup/issues/137#issuecomment-369716654, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAEEn3PqzUIQoJ50TRac2tFlu9xZ47fks5taFaegaJpZM4KlpAg .

-- Simon

[jayfk]

Valid points @bloodearnest.

The thing is, to make this really useful, pyup has to know about the full dependency tree. What if one of your transitive dependencies still relies on an older release of requests?

You could signal that your library is compatible with the latest release of requests but since pip decides what it installs you can't even test against that release unless you install it manually.

It's a difficult problem to solve, that's why I'm pushing on a PyPi build service. If someone is interested, here are some details on that: https://mail.python.org/pipermail/distutils-sig/2018-February/031962.html

[keimlink]

Why don't you use requests>=2.10.0 and only define the top end if you have to, e.g. there are breaking changes in the dependency?

[keimlink]

@kxepal has implemented support for setup.cfg in pyupio/dparse#21! 🎉 That looks like a good way to define project dependencies.

[cooperlees]

So, I couldn't decide which was the "best" issue to comment on. requests pin urllib3 (at the moment <1.25) and pyup.io continues to try and update urllib3.

• Example: https://github.com/pypa/bandersnatch/pull/224

Is there anything we could do in the Bandersnatch repo to tell pyup.io to stop with urllib3 for now?

[ntravis]

@cooperlees

Is there anything we could do in the Bandersnatch repo to tell pyup.io to stop with urllib3 for now?

https://pyup.io/docs/bot/filter/

[cooperlees]

@cooperlees

Is there anything we could do in the Bandersnatch repo to tell pyup.io to stop with urllib3 for now?

https://pyup.io/docs/bot/filter/

Thanks! I missed that and only looked @ configuration on the site.

[Querela]

For those who don't want to wait, I have implement a small workaround to allow PyUP to process my setup.py file.

Please have a look to this repository: https://github.com/DEKHTIARJonathan/keras-datasets

Basically, you define txt files which contains your dependencies and you import them inside the setup.py file as a list. Done in 5 mins and 100% efficient ;)

Function used to import any requirements.txt file:

def req_file(filename):
    with open(filename) as f:
        content = f.readlines()
    # you may also want to remove whitespace characters like `\n` at the end of each line
    return [x.strip() for x in content] 

Which gives the following inside setup.py:

setup(
    name=__package_name__,
    install_requires=req_file("requirements_lib.txt"),
    extras_require={
        'dev': req_file("requirements_dev.txt"),
        'travis': req_file("requirements_travis.txt")
    }
)

Enjoy !

I know this is late but just FYI. I was trying to work like this but ... This method described above will unfortunately not always work, e. g. you can specify -f <url> to search for packages located on this url index page (which is more or less required for some pytorch versions), in the requirements.txt but if this appears in setup.py it will break because it has an extra field in the setup(...) for this. Also, you can recursively include other requirements.txt files using -r file.txt which also will not work in the setup.py. (I was trying to use this fro extra dependencies like docs or dev.)