liormizr / s3path

s3path is a pathlib extension for AWS S3 Service
Apache License 2.0
206 stars 39 forks source link

copy files between S3 locations using S3Path #98

Closed lpravda closed 5 months ago

lpravda commented 2 years ago

Hi @liormizr, perhaps I have just missed it, but is there any way one can copy a file from the S3Path location to the S3Path location? I'm aware of S3Path.rename(), but this is not what I really want. As a workaround, I could think of

src = S3path('/src/dir')
dst - S3path('/dst/dir')

 for each in src.rglob("*"):
        if each.is_file():
            with each.open("rb") as fp:
                (dst / each.relative_to(src)).write_bytes(fp.read())

But that seems like a big overkill. I could directly use boto3, but I wondered if there is a direct support.

Thanks a lot!

four43 commented 1 year ago

My #88 is from the same motivation. We're kind of stuck between a rock and a hard place there

liormizr commented 1 year ago

Hi @lpravda Thanks for the issue The issue is that we are trying to stay as close as possible to pathlib interface So users can have the same methods and for example in prod use S3Path and in tests use local Path Not pathlib don't have a copy method. thats way we didn't implement something in the library

How ever if we will see a lot of requests we can add it. It won’t help for s3 <-> local cases but for s3 to s3 it can help.

I personally less like this approach as long that pathlib won't do the same for local file systems.