search

scrapinghub / scrapinghub-stack-scrapy

Software stack with latest Scrapy and updated deps
https://support.scrapinghub.com/support/solutions/articles/22000200402
BSD 3-Clause "New" or "Revised" License
62 stars 20 forks source link

Old version requirements causing dependency conflicts #141

Closed ryonlife closed 4 years ago

ryonlife commented 4 years ago

I ran into a number of versioning conflicts between my requirements.txt and https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-2.0/requirements.txt, when deploying my project to scrapy cloud. Specifically, awscli==1.18.17 and slackclient==1.3.2 are old, and they have dependencies locked to old versions, which conflicted with my own dependencies. Rather than build my own docker image, I just added the most recent versions of awscli and slackclient to my project's requirements.txt and was able to deploy. This seems a bit gross, and hilarious as it pertains to slackclient, but it got the job done. I suggest performing an audit of your requirements and their dependencies in order to update everything possible to the latest stable releases.

ryonlife commented 4 years ago

@vshlapakov thanks so much for the fix made at https://github.com/scrapinghub/scrapinghub-stack-scrapy/commit/544f27947859441ae306a4e45246cbbb51dd0f75. Any idea when the docker image used by shub deploy will be updated?

vshlapakov commented 4 years ago

Thanks for the report @ryonlife. Updated versions of 2.0 and 2.1 stacks are released, feel free to test it and please let us know if there's anything wrong, sorry for the inconvenience!

ryonlife commented 4 years ago

@vshlapakov I just gave it a shot with stack: scrapy:2.1 and stack: scrapy:2.1-latest and am getting the same error, which seems to indicate the dockerhub image hasn't been updated:

awscli 1.18.64 has requirement botocore==1.16.14, but you have botocore 1.16.15.

ryonlife commented 4 years ago

Ah, wait, never mind! The image is running the updated version of awscli. I just need to downgrade my botocore.