Specific versions of dependencies in setup.py

[pedersen]

This issue existed in Trac. The original can be viewed at dependencies

This issue existed on SourceForge. The original can be viewed at https://sourceforge.net/p/turbogears2/tickets/142

[pedersen]

Original Author: nickpholden, Original Timestamp: 2012-03-14 16:39:54.152000

Original Body: Please include specific version numbers for dependencies inside the turbogears setup.py. Every time I try and upgrade tg I get a different error:

pkg_resources.VersionConflict: (WebOb 1.0.8 (/home/nick/venv/talon_utils/lib/python2.7/site-packages), Requirement.parse('WebOb>=1.1.1'))

Using the tg repo isn't really an option (with custom dependancies in your own project not being in the tg repo):

pip install --index-url http://tg.gy/current/index/ -e . . . . No distributions at all found for cdecimal (from mywebapp==0.0.0.0dev)

This has been a constant problem for myself and everyone I have known who uses TG.

I can understand that you might not want to include specific versions in case someone wants to upgrade individual packages within tg (I've never spoken to this person). In this case can I suggest adding another package on pypi turbogears2-withversions so we can get installation like with any other modern web framwork.

Thanks!

[pedersen]

wave You're speaking to one of those people right now.

Where I work, we have an application that's been built for several years, but never upgraded the base packages. As such, we need to carefully work through the upgrade process. That means, in part, upgrading each individual package, testing the result, and then moving onto the next.

If we could not do that, then the process would be made more difficult for us. We would be unable to easily tell what upgrade broke our code.

On TG itself, we have also seen versions get lost: A version of a package was listed as a dependency, but later pulled from PyPI. That would render the code uninstallable.

What I will agree with, though, is that we need to help provide a way for this to work. I'll work on finding a way to make it more manageable, but I know that I do not have the answer yet. Any suggestions (other than specifying versions) would be welcome.

[nickroci]

Well, I have created http://pypi.python.org/pypi/TurboGears2Frozen/2.2.0 which fits my needs, and if I need the freedom to install particular versions then I can still use the normal TG as a backup.

I admit it would be a bit of a problem if someone totally removed a package from pypi. If that happened I would put the package in question on a local/use TG repo and use an --extra-index flag until a permanent fix could be found I guess.

I plan on doing point releases for it if something useful comes up, 2.2.0.1 for instance if a new version of X is really good.

I think some people (myself included) wont bother testing each upgraded dep separately, but install a whole new bunch and test that, the frozen package fits that approach. i.e., if you have a small shop with tight time constraints.

[pedersen]

That is one solution. I'm not in favor of it for one big reason: It's not kept automatically in sync with the main release. If you were to stop doing it, then the people relying on it would be unable to pick up where you left off.

Here's the issues as I see them, and I strongly want to find a solution for them. I don't have one yet. Any suggestions?

• Locking version numbers means partial upgrades are impossible. For some groups, this introduces unacceptable levels of risk.
• setuptools/pip/distribute will always go with the highest package that can be found without version locking.
• Private eggbaskets are not always feasible (while we make one for TG, that is not a reasonable answer for people making publicly accessible applications).
• Publicly accessible versions of code can be pulled at any time, making a package that depends on a specific version uninstallable.

At this point, the only option that looks like it could work is to lock the version numbers in the quickstart, and require people to update their application when they want to upgrade packages. Maybe provide some tools to help them find the packages they need to upgrade, too.

What do you think? Do you have anything else that will do better? I'm not very happy with that idea, so am really hoping so.

[nickroci]

I can see this is a tricky problem, if there is a worry about me falling under a bus and not continuing it then I don't mind giving other people access to it. May be it needs a CI server continuously installing it to check all the deps are on pypi, so someone can fix it if a pypi package goes missing? Otherwise you can still "fall back" to the standard install

I guess having a "build" option with tg so that it creates a package with fixed version numbers depending on your current setup. But that is kind of what is already provided by frozen and setup.py :) For my personally I prefer using standard tech so that tg apps work like all our other apps, having to do something different for tg compared to twisted or django is a real turn off. I'm also not the kind of user who wants to tweak versions of webob etc, I use the top level stuff. For the kind of people who do care, I'm not sure there is a solution that will be "flexible" enough.....

I think if people use the frozen package they know what they are getting themselves into, if they need a different version of something then they will have to roll their own tg deps and not use it, I said in the doc that it works for me and I expect it will not work for other people and that I will try and update it.

The main thing is I want to make sure that in X years time an app that I wrote that hasn't changed will still know what deps it needs and just work. I understand that some packages may not still be available but that is probably a better option than not having any version numbers, I tried to install egg-basket a while back - that was a real pain as you don't know what version of tg it used, what version of stuff it used and how to get it work.

I really don't know what a better alternative to a frozen style package is, I'll switch when one is found :)

[pedersen]

eggbasket isn't even what we use on the site. Actually, that's something that I should document at some point: How the entire site works, for other people to look at. Anyway, we use basketweaver. It produces something that works quite well for our needs.

Now, for the main discussion: You can at least see we're not just being jerks. We actually do have reasons, and they're tough to address all of them. The best solution (so far) has been to use our own basket, but that's not desirable for our users.

I have been thinking about this since I wrote the post above. The best compromise I can think of is to do the following:

In quickstarted apps, we list all dependencies with a specific version. Instead of just listing the dependencies for the app itself, we list all TG dependencies. That won't be an easy change to make, since I'm trying to automate the release process, but it should be doable. We also go back and help people figure out the versions for our previously released TG2 versions.

That would provide version locking for your application, while not affecting anybody else. And if you want to upgrade your application, you have control over how much and how rapidly.

It's still not ideal, but until that Do What I Mean feature is fully implemented, it's the only compromise that comes to mind that seems like it could work long term. What do you think?

[nickroci]

For me personally that is too much control, I want to install version X of tg with Y capabilities. I understand that one of the great things about tg is that it is a bunch of packages and you can upgrade them as you like to get new features or backwards compatibility, but I have never done that. I would be happy to accept you have to upgrade tg to 2.2.0.1 to get an upgraded version of an underlying package and that it may not be totally backwards compatible. May be that makes me an exception but it seems to work OK with something like django where it is a big lump of code.

It is much easier for me to upgrade tg for an app by changing the version of tgfrozen in the seutp.py rather than messing around with a custom command etc, or trying to understand the complexities of a bunch of packages you have never heard of before and don't have time to investigate :)

I guess the main issue with tgfrozen is that some packages may go off pypi, but may be you could put an extra_index bit into setup.py and go to the tg repo for them..... or something

Having said that I understand that tgfrozen probably goes against the tg philosophy so if you want I will put a note on the tg frozen page saying it wont be updated anymore and create a new package with a different name so people don't use it. But for me personally I think I have found a nice solution that works with the least fuss - it has literally solved the only issue I had with tg, which is in general really easy and quick to use except for the damn dependencies :)

Best Regards

[moschlar]

What about making the "package version compliance" testing process kind of a community effort? (I think Wordpress does this for their plugin repository, they let users report under which version the plugin worked or didn't work and display that). That way, the tg core developers could specify some kind of "default" version for the dependencies but users could also tell whether a specific version worked for them. Using some kind of web application for submission and displaying the values in a matrix or so. That data could maybe be used to just get custom index urls for any selection of package versions.

[pedersen]

I'm not convinced by the WordPress system for Python packaging. It does seem like a good idea, but then making pip/setuptools/etc use it will be problematic.

However, I did just find out about an option that I'd like to consider: https://github.com/PolicyStat/terrarium It supports Python 2.5/2.6/2.7, and could fix the issues for us quite thoroughly. It does require moving firmly into pip territory, but we can do that. There's people who'd argue we should have already done it.

I think I'll experiment with this tonight.

[pedersen]

So, this is still something that's very much on my mind. Terrarium isn't really a good solution, as it turns out. It's better than many options, but it's still not perfect. Why? Well, virtualenv is required. I know of at least one place that is avoiding using virtualenv at all, and will not accept this requirement.

However, I think I've just found a solution. Using pip, we can generate a frozen requirements file. pip also lets us add a place to find dependent versions, and specify a requirements file on the command line. In other words, we could provide an install command for you that looks like this:

pip install -r http://tg.gy/220/pip.txt mypackage

And that would then install all of the required versions that we need. Furthermore, due to this line (that will appear at the top of the currently non-existent file), it will use pypi where possible, and use our index where it's not, ensuring that even if versions go missing on pypi, the program can be successfully installed still. The line in question:

-f http://tg.gy/220/

I think this provides our answer. It allows people to upgrade individual packages. It provides a single command install that specifies all version dependencies without locking people into specific versions of individual packages. It's easily integrated into the build process.

What do you all think? Should I do this? Shouldn't take more than a few days to get all versions from 2.0 until current.

[pedersen]

I just had to add this: It's even extensible to other projects. Check out the documentation for freezing requirements with pip: http://www.pip-installer.org/en/latest/requirements.html#freezing-requirements

You can generate your own frozen requirements file, using the TG one as a base (since we'd serve it as text, you could even get it via web browser). This would allow you to get your packages exactly at the versions you need, and know they're safely maintained.

I've even figured out how to implement it over the course of a couple of hours (not days, I could do it over the course of one night with some automation).

It's possible I'm missing something, but this solution is the best one I've found so far. Very much needing feedback.

[moschlar]

+1

For extending the package list, this is additionally helpful: http://www.pip-installer.org/en/latest/requirements.html#recursive-requirements

[nickroci]

If you aren't going to put the deps for each version in the setup.py (and I can see the issue with changing how that works) then this seems like the best solution

[nickroci]

Having said that it seems very similar to the tg frozen approach, just with a requirements file rather than a separate setup.py ;) Personally I like the tg frozen/normal tg package approach as it means it's totally compatible with other package installations, i.e., you don't have to change you installation method just for tg. Standards FTW (not sure how standard requirements files are)?

[nickroci]

For instance, I have a CI server, I don't want to have to build/test TG apps in different ways to all the other non tg apps I have

[nickroci]

May be people would go and configure their tg build environments manually though so that may not be an issue for most people I guess

[pedersen]

Actually, though similar to the setup.py approach, it has two critical differences: It won't cause breaks if you do decide to upgrade a specific package, and integration with the build/release process is much easier (and less error-prone).

For instance, suppose a new version of SQLAlchemy comes out, and it's got a feature that you have been needing for a while now. If you install it, and the version is locked in setup.py, your application will refuse to even start, citing a version conflict. You'll have to deal with modifying setup.py, and getting all the other pieces in place, before this will work for you. With the requirements file, you can upgrade that one component, and that's it.

It's also much easier to integrate into our build/release process. A final step becomes "make new virtualenv, pip install tg2, pip freeze", and then putting that file into our basket. People can use it easily (if you're using virtualenv, which you really should be, then you've already got pip installed), so they get the benefit. And we don't have the error prone process of modifying setup.py, ensuring it didn't break, and then releasing that new version (did we remember to commit it?).

I'll put this together tonight, so we can start testing it for people to make sure I got it all correct.

[nickroci]

Well for my purposes the build and release process is the opposite - where I want what is in the virtualenv to be defined at "install" time, currently from tg frozen in the setup.py rather than the virtualenv defining what dependencies there are.

Also I do not want people to have the ability to modify the packages in a particular deploy without changing the setup.py, so the thing not starting up when the underlying packages do not match is actually an advantage for me. So people don't go and hack deployed virtualenvs on servers without changing the definition of what should be in the virtualenv (in the setup.py), meaning that we have a consistent install across machines.

If I wanted to have the flexibility to change what packages are in a tg install I would use the standard tg package and define all the deps in the setup.py. If you notice I specifically did not put a version in for sqlalchemy for tg frozen and left out mysql python.

For people who want more flexibility I guess the freeze file will work better, but then what is the point in releasing a freeze file if people are going to ignore it and install whatever versions of the underlying packages anyway :) May be it could be a useful starting point and they could fiddle afterwards, but personally I think encouraging that kind of fragmentation between installs is not that useful. For instance I wouldn't have an issue with waiting for a new version of tg to get an upgrade/bug fix for package X, that is kind of what happens anyway as you are meant to use the tg repo for initial installs, so if it isn't there then you are out of luck.

[pedersen]

I understand that this does not cover your use case entirely. You want (to the point of demanding) that everything is version locked, entirely, absolutely, inviolate.

Tonight, I put together a script that would generate the freeze file I spoke of earlier, and I'm going to have to ask the mailing list for help. As it turns out, the number of dependencies is surprisingly small to me (I thought it was much larger). Of those, only a very small handful are actually problematic. Most of the ones that are left over, for most projects, are done in that project's setup.py (SQLAlchemy being one such example).

After running a quick version of the process, I find that the only problem child right now is WebOb. The API did some significant changes going from 1.1 to 1.2, and we haven't completed those changes yet. That one, though, is locked in the setup.py file.

In other words, this entire issue might already be a non-issue. What packages are you fixing the versions for in the tg2frozen package you've created?

[amol-]

I actually think that due to the private index requirement removal and the small amount of dependencies in 2.3 release this issue should be greatly reduced. TurboGears2 package will only depend on 7 packages which have no version lock. This way all the dependencies are actually dependencies of the user application and can be locked by the user itself by preference.

It should be possible to also force version of TG dependencies inside the user application itself. Making in fact perfectly possible to always rely on a frozen set of packages through the application setup.py.