search

requests-cache / aiohttp-client-cache

An async persistent cache for aiohttp requests
MIT License
116 stars 20 forks source link

Feature request: Control what type of conditional request is being made #229

Open netomi opened 5 months ago

netomi commented 5 months ago

Feature description

The cache will use whatever method is supported depending on the presence of various header fields:

However, I encountered a corner-case when using conditional requests with GitHub, where using ETags leads to unwanted behavior, see https://github.com/eclipse-csi/otterdog/issues/201 for a summary of the problem.

It would be useful if in such cases you can control what type of information is used when sending such a conditional request, e.g. limit to the use of the Last-Modified field.

Use case

Being able to use stable conditional requests with the GitHub Rest API when using installation tokens that have to be rolled every hour.

Workarounds

There is no known workaround atm afaict, the cache will invalidate itself when a new token is being used.

Plan to implement

Tbh I am not sure if this really a useful feature or just adds complexity, it eventually should be fixed on GitHub side, but I wanted to raise and discuss this issue here as well.

JWCook commented 5 months ago

Interesting, I definitely wasn't aware of that behavior.

The GitHub API is a common enough use case that I'd be willing to add a workaround for that in this library. But since it may no longer be relevant if/when GitHub fixes this on their end, maybe this could be exposed as an undocumented/internal flag instead of part of the public API?

Something like:

session._disable_etag = True
netomi commented 5 months ago

I am not sure if disabling etag altogether results in a net-positive effect in the case of GitHub and changing tokens. The downside is that only Last-Modified headers are used, and these are not present for all API calls. This means that various calls do not benefit from conditional requests anymore.

A logic that would prefer Last-Modified over ETag when present would be more desirable imho. Right now the logic just uses any cache header that is present to construct the conditional request, however, if an Etag is present, and the token changed GitHub will return a fresh response. So in the special case of GitHub (until it might be fixed), if a Last-Modified header is present, use that to construct a conditional request, otherwise, use the ETag as a best effort.

That could work imho, would need to test how well GitHub handles If-Modified-Since headers.

JWCook commented 5 months ago

Got it, that makes sense to me. Would you like to put together a PR for that?

netomi commented 5 months ago

yeah, i can take a look and prepare a PR