search

gomods / athens

A Go module datastore and proxy
https://docs.gomods.io
MIT License
4.43k stars 499 forks source link

Allow cache configurability on the mod, zip and info files #1183

Open arschles opened 5 years ago

arschles commented 5 years ago

Is your feature request related to a problem? Please describe.

These 3 endpoints are technically cachable forever because Athens is supposed to be immutable. Currently the athens.azurefd.net proxy has a caching proxy in front of it that honors the no-cache headers that Athens sends back for these endpoints.

In the past, Athens has failed to download the entire zip file, or failed to transmit it to the proxy for one reason or another. I haven't been able to fully investigate the issue so don't have intimate details on what happened.

Additionally, when the upgrade from go 1.11.x to 1.12.x happened, checksums changed. Even if I deleted all the modules in Athens's storage, they would still be cached in the proxy forever, so that action would have been worthless.

The end result is that the proxy serves broken or old data forever, unless I manually go purge it. I currently do this if I notice a problem happening over and over. When I do purge the cache, Athens experiences a huge spike in load.

Describe the solution you'd like

Rather than do the manual purge and experience the resulting surge, I'd like to be able to configure Athens to return timeouts from the Cache-Control header that it sends back to the proxy. By doing this, mod, zip and info endpoints for a given module would periodically be hit to refresh the proxy's cache.

Describe alternatives you've considered

I could write a script to periodically purge module data in the proxy's cache.

Additional context

None

jakesylvestre commented 4 years ago

+1 on this, this is especially useful when you've got athens configured with cloudflare/basic auth. Is there any reason not to do this by default?

jakesylvestre commented 4 years ago

By the way - I put this behind @cloudflare and wanted to share the results b/c they're absolutely insane. So I added this page rule: image and the cache rate changes at my mouseover point. image

Would be great to do this w/o the page rules

jakesylvestre commented 4 years ago

In the past, Athens has failed to download the entire zip file, or failed to transmit it to the proxy for one reason or another. I haven't been able to fully investigate the issue so don't have intimate details on what happened.

@arschles by the way we haven't run into this