danielfrg / s3contents

Jupyter Notebooks in S3 - Jupyter Contents Manager implementation
Apache License 2.0
248 stars 88 forks source link

using this project with pip-compile fails to find compatible dependencies #135

Open gvacaliuc opened 2 years ago

gvacaliuc commented 2 years ago

Hi! Thanks for the awesome library!

I'm trying to use this in a larger project w/ pip-compile from pip-tools, but I'm running into an issue finding compatible dependencies:

$ mkdir tmp
$ cat > tmp/requirements.in <<EOF
> s3contents
> EOF
$ pip-compile tmp/requirements.in 
Could not find a version that matches aiobotocore[boto3]<2.0.0,>=1.4.0,~=2.0.1 (from s3contents==0.9.0->-r tmp/requirements.in (line 1))
Tried: 0.0.5, 0.0.6, 0.1.1, 0.2.0, 0.2.1, 0.2.1, 0.2.2, 0.2.2, 0.2.3, 0.2.3, 0.3.0, 0.3.0, 0.3.1, 0.3.1, 0.3.2, 0.3.2, 0.3.3, 0.3.3, 0.4.0, 0.4.0, 0.4.1, 0.4.1, 0.4.2, 0.4.2, 0.4.3, 0.4.3, 0.4.4, 0.4.4, 0.4.5, 0.4.5, 0.5.0, 0.5.0, 0.5.1, 0.5.1, 0.5.2, 0.5.2, 0.5.3, 0.5.3, 0.6.0, 0.6.0, 0.7.0, 0.7.0, 0.8.0, 0.8.0, 0.9.0, 0.9.1, 0.9.1, 0.9.2, 0.9.2, 0.9.3, 0.9.3, 0.9.4, 0.9.4, 0.10.0, 0.10.0, 0.10.1, 0.10.1, 0.10.2, 0.10.2, 0.10.3, 0.10.3, 0.10.4, 0.10.4, 0.11.0, 0.11.0, 0.11.1, 0.11.1, 0.12.0, 0.12.0, 1.0.0, 1.0.0, 1.0.1, 1.0.1, 1.0.2, 1.0.2, 1.0.3, 1.0.3, 1.0.4, 1.0.4, 1.0.5, 1.0.5, 1.0.6, 1.0.6, 1.0.7, 1.0.7, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.1.2, 1.1.2, 1.2.0, 1.2.1, 1.2.2, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.4.0, 1.4.1, 1.4.2, 2.0.0, 2.0.1
Skipped pre-versions: 0.6.1a0, 0.6.1a0
There are incompatible versions in the resolved dependencies:
  aiobotocore[boto3]<2.0.0,>=1.4.0 (from s3contents==0.9.0->-r tmp/requirements.in (line 1))
  aiobotocore~=2.0.1 (from s3fs==2021.11.1->s3contents==0.9.0->-r tmp/requirements.in (line 1))
$ python -V
Python 3.8.10
mrosales commented 2 years ago

I just ran into this too. It looks like the constraints need to be tweaked a bit.

s3contents specifies these dependencies

aiobotocore = {version = "^1.4.0", extras = ["boto3"]}
s3fs = ">=2021.11.0"
gcsfs = {version = "^2021.11.0", optional = true}

The issue is that s3fs upgraded to aiobotocore~=2.0.1 in 2021.11.1 which means that the s3fs constraints are exclusively satisfied by s3fs==2021.11.0.

Adding this requirement to my pip-compile input fixed the resolver which was not smart enough to figure this out, but it would be nice if s3contents could either fix the constraint on s3fs to specify the one compatible version or expand the range and drop the requirement on aiobotocore<2