Loosen the package version constraints of dependencies

scrapinghub / shub-workflow

BSD 3-Clause "New" or "Revised" License

13 stars 14 forks source link

Loosen the package version constraints of dependencies #3

Closed starrify closed 5 years ago

starrify commented 5 years ago

Currently (as of 3393bb8) this package has got a list of dependencies which is in conflict to some other commonly required ones, like those required in scrapinghub-stack-scrapy:1.6-py3.

It may be resolved by overriding dependencies from the base docker image (example). However, it'll be good if we just loosen the constraints (if possible) here.

kalessin commented 5 years ago

As I can see, the only difference is in pyyaml. Did you notice another conflict?

starrify commented 5 years ago

As suggested in the earlier example, we've got also botocore and awscli to care about.

Sample dependency chains:

shub-workflow@3393bb8 -> boto3>=1.9.92 -> botocore<1.13.0,>=1.12.163
scrapinghub-stack-scrapy@1.6-py3 -> botocore==1.12.91

BTW I've not yet checked any other version of scrapinghub-stack-scrapy.

kalessin commented 5 years ago

The boto3 minimal version imposed in setup.py is the required for s3fs for working correctly. I think the problem here is on scrapinghub-stack-scrapy side. They are who should loose the required version. Also, considering that botocore is constantly releasing new versions with fixes.

starrify commented 5 years ago

Right. Much agreed! Thanks. :)

kalessin commented 5 years ago

I updated the requirement string of pyyaml

https://github.com/scrapinghub/shub-workflow/commit/9111442b47ee961776e27d4d86422b6add46add5