liormizr / s3path

s3path is a pathlib extension for AWS S3 Service
Apache License 2.0
209 stars 41 forks source link

Add ability to change AWS_PROFILE for Resource in between S3Path calls #161

Open nathanpowellcma opened 9 months ago

nathanpowellcma commented 9 months ago

Hi,

Thanks for the great package - finding it super useful! We use AWS_PROFILEs and sometimes need to switch between profiles to access different buckets. We spotted the _cleanup() method in the tests but can't use it because the way the resource is created doesn't use the Session and therefore will always use the profile at the time that boto3 is imported (or, at least, this is my understanding). The following code snippet works if you change default_resource to return boto3.Session().resource('s3').

# %%
from s3path import accessor, S3Path
import os

# %%
def _cleanup():
    accessor.configuration_map.get_configuration.cache_clear()
    accessor.configuration_map.get_general_options.cache_clear()
    accessor.configuration_map.is_setup = False

# %%
os.environ["AWS_PROFILE"] = "shared1"
test_path = S3Path("/data1/test.txt")
test_path.read_text()

# %%
_cleanup()

# %%
os.environ["AWS_PROFILE"] = "shared2"
test_path1 = S3Path("/data2/test.txt")
test_path1.write_text()

I've made a PR which passes the tests. I'm assuming that you wouldn't want to provide some sort of cache clearing / cleanup utility to the users, but I'm happy to implement it myself given that it's relatively easy.

liormizr commented 8 months ago

Hi @nathanpowellcma I added a comment in your PR Thanks