search

requests-cache / aiohttp-client-cache

An async persistent cache for aiohttp requests
MIT License
121 stars 21 forks source link

HTTP responses larger than 400kB are not handeled by DynamoDBBackend #177

Closed domin4815 closed 1 year ago

domin4815 commented 1 year ago

The problem

DynamoDBBackend raises an exception when the HTTP response size is greater than 400kB. DynamoDB has a size limit of 400kB for an entity stored in the table.

Expected behavior

The cache could skip the HTTP response instead of trying to put it in the DynamoDB table. Another way would be to somehow split the response and put it as multiple entities.

Steps to reproduce the behavior

Try to perform HTTP GET on https://microsoftedge.github.io/Demos/json-dummy-data/5MB.json with DynamoDBBackend.

Workarounds

We can probably define a filter function for it so the heavy responses are not cached.

Environment

JWCook commented 1 year ago

I agree, I think skipping the cache would be better than raising an exception in this case. I'll get that fixed soon.

Also related: https://github.com/requests-cache/requests-cache/issues/579 I probably don't have time to implement a secondary storage backend for large responses right now, but it's definitely something I've considered.

JWCook commented 1 year ago

These changes are now available in v0.10.