Cannot configure default session

[Alexndrrr]

docs/advance.rst says to call boto3.setup_default_session to override configuration options.

However, that only works if that call to boto3.setup_default_session occurs before s3path is imported, because the _S3Accessor object is created at module import time, and that's when it creates the session using default parameters.

The (very hacky) workaround I came up with is to call S3Path._accessor.__init__() after I reconfigure the default session, which seems to work but is not pretty.

Not sure what the ideal solution is to this—perhaps a module-level "set session" function?

[liormizr]

Hi @Alexndrrr ,

I think that we fixed it in the last version. Please see issue #87

can you confirm that you are working on the latest version? (Version: 0.3.3)

[Alexndrrr]

Thanks for your help @liormizr ! I can confirm I was having this issue with version 0.3.2. In the meantime I switched to 0.3.3 and am still having troubles that seem to indicate this issue is not completely resolved.

My app is checking to see if the user's existing credentials are functional to hopefully skip the login process whenever possible. My app looks like this:

os.environ['AWS_PROFILE'] = ...
from s3path import S3Path
try:
    next(S3Path(...).iterdir())
except botocore.exceptions.ClientError:
    login_to_aws()

next(S3Path(...).iterdir()) # This should always succeed now

In this order the app works fine. But if I import S3Path before setting AWS_PROFILE, it fails with ClientError: An error occurred (ExpiredToken) when calling the ListObjectV2 operation: The provided token has expired.

[Dobatymo]

I am on version 0.3.4 and still get NoCredentialsError errors even after calling boto3.setup_default_session. I work around by setting

from s3path import S3Path, _S3Accessor
path = S3Path.from_uri("s3://...")
path._accessor = _S3Accessor(aws_access_key_id=..., aws_secret_access_key=...)

[liormizr]

Hi @Dobatymo

I don't see this issue reproducing in my env

Do you have the same behaviour like @Alexndrrr mention? (with the profile setting up?)

[Dobatymo]

@liormizr it should be the same behaviour as @Alexndrrr's. Reading the code it does make sense that it doesn't work as the _S3Accessor is initialized at import time and it's not lazy enough. https://github.com/liormizr/s3path/blob/6d96ff276c89d23af9941044d3c31811c81cb093/s3path.py#L555

[liormizr]

@Dobatymo if you looked at the code and if I understand you right, Would changing the Accessor configuration_map to be lazy will solve the issue?

https://github.com/liormizr/s3path/blob/6d96ff276c89d23af9941044d3c31811c81cb093/s3path.py#L158

[Dobatymo]

@liormizr Hi sorry please let me test more. I can replicate it in my first environment (databricks), but not in my second (local). So it might be an issue with the dependencies.

EDIT: Please disregard all my comments. The fix for https://github.com/liormizr/s3path/issues/87 is working as intended for me. My boto3 dependency was incompatible with my botocore version. That caused the issues for me.

The lesson here for me is that installing awscli will upgrade the botocore dependency, but not boto3...