search

liormizr / s3path

s3path is a pathlib extension for AWS S3 Service
Apache License 2.0
206 stars 39 forks source link

Make boto3 dependency optional/soft for cases where only PureS3Path is needed #122

Open echernyavskiy opened 1 year ago

echernyavskiy commented 1 year ago

Looking at the code it seems PureS3Path does not need anything from boto3. However boto3 is a hard requirement for s3path. This is problematic for environments like AWS Lambda Python runtime that has boto3 "built-in". Could boto3 be a soft dependency like awacs for troposphere for example?

liormizr commented 1 year ago

Hi @echernyavskiy Thanks for the issue :-)

I don't full understand your use case in AWS Lambda Python like you say you already have boto3 built-in So when you do import from s3path you don't have an issue. You want to install s3path without "reinstalling" boto3 again? is this your use case?

echernyavskiy commented 1 year ago

@liormizr, right - since boto3 is "built into" the runtime, I don't have to define it as a dependency in my pyproject.toml / requirements.txt. Except s3path requires it, so it gets installed into the venv anyway. Could it be a soft dependency instead?

liormizr commented 1 year ago

Hi @echernyavskiy, Sorry for the delay.

We have an issue to move boto3 from the dependencies. Basically it means that all the users of this package will need to do change the way that thy install the package and that is a big change.

Now I don't see a nice way to exclude dependency. If you have an idea how to do so I'm open for suggestions Also I see that AWS do recommend to specify your boto3 version so there won't be surprises if AWS decide to change boto3 version.

plondino commented 1 month ago

There does seem to be a lively discussion about this here: https://discuss.python.org/t/optional-dependency-groups-omitting-package-requirements/20833/1 - I agree that this would be useful. I have some libraries that need to manipulate both local filesystem paths and S3 paths, and the PureS3Path functionality would be extremely useful, but adding the full boto3 library as a required dependency is a non-starter since many consumers won't even use the AWS functionality.

liormizr commented 1 month ago

@plondino

Boto3 is currently a hard requirement
How ever in case you are using only PureS3Path it's not going to be loaded. We changed it so it will be lazy imported only when using S3Path.