search

openzim / python-scraperlib

Collection of Python code to re-use across Python-based scrapers
GNU General Public License v3.0
18 stars 16 forks source link

Dependency resolve error with urllib3 when using in combination with kiwixstorage #74

Closed satyamtg closed 3 years ago

satyamtg commented 3 years ago

Since we use a range for the requests package, urllib3 1.26 gets installed now. However, botocore in kiwixstorage requires version < 1.26 and that causes issues while running a scraper.

chandra20sekhar commented 3 years ago

Hi ,I would like to work on this issue .I am a new beginner ,and interested in improving python scrappers project .Can you suggest any ways to get started about the project and this issue

rgaudin commented 3 years ago

Sorry for not responding earlier. I faced it yesterday to fix gutenberg which uses kiwixstorage as well. I don't think there's much to fix here beside updating kiwixstorage (0.5). What I did there was to allow any boto3 version bellow 2. See https://github.com/kiwix/python_storagelib/commit/09a9cbd2a99a366f58e59e87ec0f93c03fc6f39b They don't seem to have much rules regarding versioning and they are including git of botocore. I believe this relaxed versions for boto3 combined with our flexible (<3) for requests (which uses proper versioning) should solve the problem for some time.