Cannot find a working version for S3FS for AWS Lambda

[alexanderluiscampino]

Since last night, all my lambdas running with S3FS start failing. Trying now to find a suitable combination of versions is leading me nowhere.

The following pinned libs on my lambda, are failing due the reason below:

yarl 1.7.2
wrapt 1.14.0
typing-extensions 4.1.1
tenacity 8.0.1
s3fs 2022.2.0
requests 2.25.1
pyrsistent 0.17.3
pydantic 1.8.2
multidict 6.0.2
lambdutil 0.4.24
lambda-decorators 0.6.0
jsonschema 3.2.0
jsonpickle 2.0.0
jeffy 1.1.1
idna 3.3
fsspec 2022.2.0
frozenlist 1.3.0
fastjsonschema 2.15.1
email-validator 1.1.2
dnspython 2.1.0
charset-normalizer 2.0.12
chardet 4.0.0
certifi 2021.5.30
aws-xray-sdk 2.8.0
aws-lambda-powertools 1.22.0
attrs 21.4.0
async-timeout 4.0.2
aiosignal 1.2.0
aioitertools 0.10.0
aiohttp 3.8.1
aiobotocore 2.1.2
urllib3 1.26.6
six 1.16.0
s3transfer 0.5.2
python-dateutil 2.8.2
jmespath 0.10.0
botocore 1.23.32
boto3 1.20.32
setuptools 56.0.0
rapid-client 0.0.0
pip 21.1.1

{
  "errorMessage": "Unable to import module 'orchestrator/handlers/routine_metadata': cannot import name 'is_valid_ipv6_endpoint_url' from 'botocore.endpoint' (/var/runtime/botocore/endpoint.py)",
  "errorType": "Runtime.ImportModuleError",
  "stackTrace": []
}

[martindurant]

What is s3fs-test? That's the thing that poetry is complaining about. To be sure, there was no new release of s3fs in the last week, whereas aiobiotocore releases all the time. I cannot tell what might have changed for you!

[alexanderluiscampino]

s3fs-test is just the name of my repo, it doesn't affect anything.

I understand s3fs didn't get released, but AWS updated their boto+botocore versions

botocore 1.23.32
boto3 1.20.32

which made aibotocore fail, which s3fs is dependent on. Is this a aibotocore issue? I'm just wondering if S3Fs should be dependent on a such a fickle library...

[martindurant]

I asked because the output of poetry suggests that the conflicting requirement on aiobotocore comes from there.

aiobotocore 2.1.2

aiobotocore always has a very strict pin on botocore (check their release notes), and s3fs purposefully doesn't use the latest available aiobotocore, since problems upstream are common.

[alexanderluiscampino]

100% agree with you on this. But what I'm more hinting at is what kind of initiatives can we have that keeps s3fs in sync with latest botocore releases. Reason I ask is the following. I rely heavily on s3fs in production on AWS Lambdas. It serves a great API to interact with S3. Botocore is shipped by default with all Lambdas and we are at the mercy of when AWS decides to ship a new version of it. That happened Monday night when they went from 1.20.x -> 1.23.32 which created a breaking change on aibotocore which cascaded down to s3fs.

The solution was to ship my s3fs layer with a pinned version of boto3 and botocore. But this adds a lot of bloat to the lambdas -> cold starts amongst other issues.

[martindurant]

You should cross-post with aiobotocore, because installing any version of that ought to make sure that you get a compatible botocore, even if it needs to be downgraded. s3fs could also pin to botocore, but we obviously can't use any aiobotocore that is not released yet. If their pin is not working, then neither will ours!

[alexanderluiscampino]

Will post there as well, thank you Martin!