Snakebasket choosing the latest version of a dependency is an anti-feature.

[mrmachine]

In your README you say foo depends on ReportLab=1.7 and bar depends on ReportLab=1.9, causing dependency hell. Snakebasket resolves this by installing the latest of the two versions and assuming that it will work for foo.

This example itself is only causing dependency hell because it demonstrates a misuse of install_requires, which should not be used to pin an exact version to a package's dependencies. install_requires should be used to exclude versions that are known not to work.

This is why generally dependencies that are defined with install_requires have a minimum version. Usually we don't know that a future package won't be compatible, so we assume that future packages are compatible until they are released and prove otherwise.

At such a time, you can then update the install_requires for your package to exclude additional versions of its dependencies which are known to not work. For example, if a new feature was added to ReportLab 1.7 that your package requires, but compatibility was broken in 1.9 and then fixed in 1.11, you could have ReportLab>=1.7,<1.9,>=1.11 or ReportLab>=1.7,!=1.9,!=1.10.

Correct use of install_requires should drastically reduce the possibility of dependency hell. If one package truly and legitimately requires exactly ReportLab==1.7 and another requires exactly ReportLabl==1.9, then this is something that Snakebasket won't be able to solve with an assumption.

The point of pip and requirements.txt is to then freeze all the requirements of your project and all of its dependencies, so that you can test and reproduce exactly the same environment reliably. So you should only be using setuptools recursive dependency handling (e.g. install_requires) once (so that you can freeze the requirements), or perhaps a second time when you want to try and start fresh and see if your project will run with the latest versions of all its dependencies.

So I guess the point of this issue is -- can you remove this behaviour from Snakebasket? Or at least make it optional, and ideally not the default?

The reason I am interested in using Snakebasket is for its recursive processing of requirements.txt, which IS quite useful in some circumstances. My use case is probably quite similar to yours. We have many private utility apps that are hosted in git repositories. These are not released to PyPI and they are often under heavy development.

I am looking for a way that we can define a project dependency on a set of core utility apps that are used by the project, and have the dependencies for those utility apps be installed automatically. For example:

• project depends on lib-a and lib-b.
• lib-a depends on lib-c and lib-d.

All of the above would be unpinned, e.g. pointing to the @master branch of a git repository. I'd do sb install -r requirements-unpinned.txt initially. Then I would do pip freeze > requirements.txt so that the entire environment can be reliably recreated.

Then it is up to the person developing the project and its collection of dependencies to resolve any conflicts and re-freeze the entire dependency tree.

[neumark]

Hi

Sorry for taking a while to get back to you! The requirements.txt format AFAIK doesn't allow one to specify version ranges (only >= a particular version). You are absolutely correct that using install_requires in setup.py would give us finer grained control over our dependencies, but that format doesn't enable us to specify git URLs.

It"s possible that either through new improvements to pip or simply my own misunderstanding it is possible to somehow get version ranges and git URLs into the same file (requirements.txt or setup.py). If this is the case, could you send me an example which does so?

Thanks and all the best, Pete

[mrmachine]

You are right that requirements.txt are intended to freeze an absolute version for your entire dependency tree. I'm not sure if it is possible to specify a range in requirements.txt, but even if you could I imagine that would not work for editable git clones.

However, in your examples you are pinning absolute versions in setup.py. Generally you should use setup.py to define which packages are required, and which versions of those packages are NOT compatible, as loosely as possible. Then once installed, you would freeze the absolute versions for the entire dependency tree in requirements.txt.

Setuptools doesn't understand editable git clones, which is why I would like to use SnakeBasket to recursively install dependencies listed in in requirements.txt. This should have nothing to do with dependencies that are defined in setup.py.

In my case, all I want to do is have a requirements.txt for each private package we host on GitHub, which installs editable git clones at the master branch (or some other branch) for each package's dependencies. Then I would still use pip to freeze absolute versions for the entire dependency tree to requirements-frozen.txt.

In your case, perhaps using git branches would also be sufficient instead of using version ranges. Then it should be up to the person doing deployment to still freeze the absolute versions for the dependency tree.

[mrmachine]

Perhaps more succinctly: Re-usable packages (the ones being installed recursively) should not use absolute versions in either requirements.txt or setup.py. Both should list only the package name, and if necessary for setup.py also narrowly exclude versions that are known not to work. For editable git clones in requirements.txt, use a branch name.

Then there is no need for SnakeBasket to do any version conflict resolution. It is up to the project developer (who is installing the re-usable dependencies) to manually resolve those conflicts when they freeze a flattened dependency tree with absolute version numbers.

[neumark]

Thanks for the clarification.

As you probably know, snakebasket is actually a wrapper around pip code, which does most of the heavy lifting. In pip, it's pretty difficult to have a package's name come from requirements.txt and version number range come from setup.py. The reason is that setup.py is not processed by pip: it's actually processed directly by setuptools which has no idea of pip or snakebasket.

I'm sure there's a way to do what you suggest, and we even considered it, but it looked like a tough road to take.

What you can do, is to have library dependencies reference only package names, eg: libfoo depends on ReportLab Applications would then specify the minimum version: foo-app depends on libfoo and ReportLab==0.16.1

If you pass --prefer-pinned-revision to sb, then ReportLab 0.16.1 would be installed. I know this is not perfect, because you still can't give version ranges. This is something which we also suffer from occasionally (and hack around right now by removing packages after the sb run and reinstalling a specific version with sb or pip).

I think the longterm solution would be to replace the requirements.txt format with a json file which would then enable more control over dependencies (eg: adding version ranges).

Supporting git in setup.py would be great, but that means patching setuptools which is a beast (actually a beast forked several times, each fork with wildly differing os-dependant logic in various versions).

[mrmachine]

I'm not suggesting a fork to pip or setuptools, or adding support for editable git repositories in setuptools (it already supports git URLs via dependency_links, just not editable), or a new file format for requirements.

I'm suggesting that SnakeBasket should not be needed for dependency version conflict resolution, if dependencies are specified abstractly (no version or commit hash) in setup.py and requirements.txt.

For re-usable libraries and your project:

• In setup.py you should only list abstract dependencies (no version number, except to exclude those you know are broken) that are hosted publicly.
• In requirements.txt you should only list abstract dependencies (e.g. no commit hash for editable VCS URLs). These can be hosted privately.

When you do sb install -r requirements.txt:

• SnakeBasket downloads each editable requirement (git, svn, etc.)
• SnakeBasket runs sb install -r <path-to-requirements.txt> for each downloaded editable requirement. This recurses until there is no requirements.txt file for a downloaded editable requirement.
• SnakeBasket runs pip install -e <path-to-package> for each downloaded editable requirement. This happens after its requirements.txt file has been processed.

This should recursively install the latest version of each editable requirement before invoking setuptools for that requirement. If setuptools defines an abstract dependency in install_requires, it will have already been installed.

If a re-usable library doesn't work with the latest version of one of its dependencies, add a branch to the dependency that is compatible with the library and add it to the VCS link.

Then, it is up to the project developer to freeze the entire flattened dependency tree into requirements-frozen.txt and manually resolve any problems. This can be installed with pip install --no-deps -r requirements-frozen.txt to recreate exactly the same environment.

The point is, regardless of whether or not you are using setup.py or requirements.txt, if you are defining dependencies for a re-usable app or library, you should only exclude versions that are known not to work (for setup.py) or use an editable VCS URL that points to the default branch (or a different branch if the default is known not to work) and not to a specific commit (for requirements.txt).

Only the project developer should resolve version conflicts and define hard dependencies (with a version or commit hash) in requirements.txt.

[neumark]

Hey,

Snakebasket is a an extended version of pip. Forking / patching setuptools is beyond it's scope. That's not to say I disagree with you. I would also prefer to have version ranges in setup.py, but that's beyond what snakebasket was designed to do.

On Mon, Jun 16, 2014 at 9:14 AM, Tai Lee notifications@github.com wrote:

I'm not suggesting a fork to pip or setuptools, or adding support for editable git repositories in setuptools (it already supports git URLs via dependency_links, just not editable), or a new file format for requirements.

I'm suggesting that SnakeBasket should not be needed for dependency version conflict resolution, if dependencies are specified abstractly (no version or commit hash) in setup.py and requirements.txt.

For re-usable libraries and your project:

• In setup.py you should only list abstract dependencies (no version number, except to exclude those you know are broken) that are hosted publicly.
• In requirements.txt you should only list abstract dependencies (e.g. no commit hash for editable VCS URLs). These can be hosted privately.

When you do sb install -r requirements.txt:

• SnakeBasket downloads each editable requirement (git, svn, etc.)
• SnakeBasket runs sb install -r for each downloaded editable requirement. This recurses until there is no requirements.txt file for a downloaded editable requirement.
• SnakeBasket runs pip install -e for each downloaded editable requirement. This happens after its requirements.txt file has been processed.

This should recursively install the latest version of each editable requirement before invoking setuptools for that requirement. If setuptools defines an abstract dependency in install_requires, it will have already been installed.

If a re-usable library doesn't work with the latest version of one of its dependencies, add a branch to the dependency that is compatible with the library and add it to the VCS link.

Then, it is up to the project developer to freeze the entire flattened dependency tree into requirements-frozen.txt and manually resolve any problems. This can be installed with pip install --no-deps -r requirements-frozen.txt to recreate exactly the same environment.

The point is, regardless of whether or not you are using setup.py or requirements.txt, if you are defining dependencies for a re-usable app or library, you should only exclude versions that are known not to work (for setup.py) or use an editable VCS URL that points to the default branch (or a different branch if the default is known not to work) and not to a specific commit (for requirements.txt).

Only the project developer should resolve version conflicts and define hard dependencies (with a version or commit hash) in requirements.txt.

— Reply to this email directly or view it on GitHub https://github.com/prezi/snakebasket/issues/15#issuecomment-46147383.

Peter Neumark DevOps guy @Prezi http://prezi.com

[mrmachine]

Confused. I'm not asking for version ranges or forking / patching setuptools? I'm just asking for a way to disable SnakeBasket's version conflict resolution "feature".

I don't want SnakeBasket making those decisions automatically. We won't use pinned dependency versions in re-usable library apps (as described above). We will use flattened and pinned dependencies only in projects, and it is up to the project developer to resolve conflicts.

We just want SnakeBasket to recursively install editable dependencies from requirements.txt.

[neumark]

Hi Tai,

This is not something we plan on adding ourselves, but if you want to write a patch, this is where I'd look in the code: https://github.com/prezi/snakebasket/blob/master/snakebasket/commands/install.py#L250

Good luck, Peter

On Mon, Jun 16, 2014 at 9:53 AM, Tai Lee notifications@github.com wrote:

Confused. I'm not asking for version ranges or forking / patching setuptools? I'm just asking for a way to disable SnakeBasket's version conflict resolution "feature".

I don't want SnakeBasket making those decisions automatically. We won't use pinned dependency versions in re-usable library apps (as described above). We will use flattened and pinned dependencies only in projects, and it is up to the project developer to resolve conflicts.

We just want SnakeBasket to recursively install editable dependencies from requirements.txt.

— Reply to this email directly or view it on GitHub https://github.com/prezi/snakebasket/issues/15#issuecomment-46150030.

Peter Neumark DevOps guy @Prezi http://prezi.com