search

fsspec / s3fs

S3 Filesystem
http://s3fs.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
836 stars 268 forks source link

Add transfer configuration to support concurrent downloading #841

Open ser-amzn opened 5 months ago

ser-amzn commented 5 months ago

I am trying to speed up downloading of large files from S3.

I can enable parallel downloading of file parts in boto3 using https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3.html:

    config = TransferConfig(
        multipart_threshold=1000000000,
        multipart_chunksize=50000000,
        max_concurrency=128,
    )
    s3_client.download_file(bucket, key, dst, Config=config)

Can I do the same in s3fs? if seems that passing s3_additional_kwargs doesn't help:

s3 = s3fs.S3FileSystem(anon=False, s3_additional_kwargs={'Config': config})

It would be great to have this feature in s3fs.

martindurant commented 5 months ago

I don't believe that requesting parallel streams for a single file will increase bandwidth - you should simply saturate your network bandwidth once the transfer begins. Concurrency helps most when making many requests and wishing to amortise the latency.

(decoding and compression may parallelise well, but they are not normally significant, and python's GIL makes it hard to achieve)

Having said that, s3transfer does do some clever things, so if anyone is interested in calling it within get/download[_file], I would be interested to see this.