Check requirements against requirements of all dependencies

[jayvdb]

Due to the way pip works, without a proper dependency graph, the version installed is not deterministic unless our versions are identical to the versions of our dependencies.

All requirements files should be in sync. A bear needs to check this.

[jayvdb]

See https://github.com/coala/coala/issues/3200 and https://github.com/coala/coala/issues/3183 for the types of problems that should be avoidable.

[jayvdb]

https://github.com/coala/coala/pull/3444 is another example.

[jayvdb]

https://pypi.python.org/pypi/PinRequirementsBear by @Adrianzatreanu looks interesting.

https://github.com/landscapeio/requirements-detector looks like it only goes one level deep, and doesnt catch indirect dependencies. However it has some very useful logic, which can help if recusing through dependencies.

[jayvdb]

Note I would be suspicious of any solution which installs many sets of requirements and then uses pip freeze or similar to determine the dependencies. Something like https://pypi.python.org/pypi/freeze-requirements will stablise the system somewhat, given that it is pinning all dependencies and also has a sorted list (sorted introduces stable order, which pip needs). But that means we loose the ability to accept patched and minor versions.

pip will happily install a conflicting set of packages in some conditions, and those conflicts are only visible when pkg_resources is used (such as via setuptools hooks).

The most mature approach to solving this problem appears to be https://pypi.python.org/pypi/openstack-requirements . It may look like overkill for coala ; while coala is much smaller, we have dependencies on a lot of foreign distinct python projects, and some of those projects (especially python linters) have inter-dependencies meaning the correct usable set is actually quite constrained by forces beyond our control.

One way we could release some of this pressure is to make some bears incompatible with other bears, when we know their requirements are not reconcilable. E.g. currently we have to hold back Radon, otherwise we cant have pyflakes 1.4 . https://github.com/coala/coala-bears/pull/1242

[jayvdb]

Another problem which has been hotfixed: https://github.com/coala/coala-bears/pull/1282 (and is probably the reason https://github.com/coala/coala/issues/3532 was created)

[pratyushprakash]

@jayvdb Just to be clear, the bear is required to report any mismatch between the dependencies listed in the requirements files and the dependencies of the dependencies listed in said requirements files?

[jayvdb]

Correct. Two levels deep (i.e. bear requirements vs requirements of the bears requirements) would be 'enough' to solve most of the problems coala enounters. However the final solution needs to be recursive.

On Mon, 23 Jan 2017 01:20 Pratyush Prakash, notifications@github.com wrote:

@jayvdb https://github.com/jayvdb Just to be clear, the bear is required to report any mismatch between the dependencies listed in the requirements files and the dependencies of the dependencies listed in said requirements files?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/coala/coala-bears/issues/1113#issuecomment-274348367, or mute the thread https://github.com/notifications/unsubscribe-auth/AAA69EdsxqggLQ3iVYxTmCHNWocOakTNks5rU53-gaJpZM4LJpNx .

[bkhanale]

This issue is reserved under the GSoC project Improve Generic Bear Quality.

[jayvdb]

One way to approach this is to collect them all into a single requirements file, find any conflicts within the file (this bit still needs analysis), and then that file can be used as a pip 'constraints' file for any use of pip to avoid wrong versions being installed.