search

man-group / pkglib

Company-centric Python packaging and testing library
39 stars 22 forks source link

pkglib's dependency resolution is too strict #42

Open Wilfred opened 10 years ago

Wilfred commented 10 years ago

Scenario 1

We want to install C. Currently, pkglib will set Resolver.best based on the currently installed packages: https://github.com/ahlmss/pkglib/blob/cf3299a5abdbda5d9a3be535ef96bff74fc11550/pkglib/pkglib/setuptools/buildout.py#L502 . As a result, we'll try to find a version of C that satisfies A==1.0, B==1.0 and everything works fine.

Scenario 2

We want to install C.

This won't work. Because we consider the currently installed packages, pkglib will try to find a version of C that satisfies A==2.0, B==1.0 and we can't proceed.

Scenario 3

We want to install C. This won't work (and indeed it should not).

Solutions

1: Consider everything installed to be a requirement

This is what pkglib currently does, and it fails on scenario 2.

2: Allow anything in the virtualenv to be upgraded as necessary

ahl.pkgutils used to do this (and I ended up restoring this behaviour), and I believe pkglib used to do this too. This fixes scenario 2, but breaks in scenario 1 (B's requirements are ignored, so we end up with A==2.0, B==1.0, C==1.0).

3: Allow a project to declare dependencies.

Ideally, we'd be able to communicate to pkglib that we want both B and C installed, and allow pkglib to find a set of packages that keeps everything happy (so scenarios 1 and 2 succeed) whilst still refusing to proceed in scenario 3.

Status

I'm unable to change our internal code to be based on pkglib, since I've hit scenario 2 and it makes us unable to install some packages. I'd like to change the dependency resolution logic to solution 2 (as it was before) for now.

Thoughts, @eeaston?

Wilfred commented 10 years ago

One interesting side-effect of reverting our dependency resolution logic is that we'd be using the same logic as pip. We could then replace our dependency resolver wholesale with pip's.

eeaston commented 10 years ago

Hi! Interesting case. Sorry that it's causing merge problems - I'll try and make a minimal test case to reproduce. In scenario 3, what do you mean by 'project'? I'm not sure about replacing the resolver with pip, though I haven't dug through pip's code. Do you know if it does backtracking? This and the prefer-final/prefer-dev are the main features of pkglib's resolver.