search

datalad / datalad-crawler

DataLad extension for tracking web resources as datasets
http://datalad.org
Other
5 stars 16 forks source link

Switch to boto3 #142

Open yarikoptic opened 2 weeks ago

yarikoptic commented 2 weeks ago

It is to happen for datalad core in coming minor release

/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/datalad_crawler/nodes/s3.py:21: in <module>
53
    from datalad.support.s3 import get_key_url
54
E   ImportError: cannot import name 'get_key_url' from 'datalad.support.s3' (/home/runner/work/datalad/datalad/datalad/support/s3.py)

and also there is some direct use of boto here:

datalad_crawler/nodes/s3.py:from boto.s3.key import Key
datalad_crawler/nodes/s3.py:from boto.s3.prefix import Prefix
datalad_crawler/nodes/s3.py:from boto.s3.deletemarker import DeleteMarker
setup.py:        'boto',  # used directly by datalad_crawler/nodes/s3.py

so meanwhile we would need to restrict for crawler to have datalad < 1.1.0

effigies commented 2 weeks ago

The ImportError is fixed here: https://github.com/datalad/datalad-crawler/pull/143

That doesn't mean that use of the S3Downloader is working correctly. It isn't.