Repo index can get cached for too long (by browser?)

j005u commented 1 year ago

Device

Goggles V2

Trace ID (bototm left)

No response

Describe the issue

When the repo is updated with a new version of a package, but the user has an ongoing browser session, the repo index will fail to cleanly redownload even on a configurator refresh.

This results in the user getting 404 errors, because the previous .ipk is no longer available.

A quick workaround is to eg. open fpv.wtf in Edge instead of Chrome.

I suspect, but am not sure, this is due to the browser caching the repo index. While it's entirely workaroundable, technically this does break installs for X amount of time each time we update the repo index with a new package version. Long term, we should probably solve it.

j005u commented 1 year ago

Note: package URLs should not get re-used, meaning we should be able to safely cache them.

This really only applies to Package(z) or any other possible static URLs where the contents changes.

benlumley commented 1 year ago

Just had a dig into this...

I suspect, but am not sure, this is due to the browser caching the repo index. While it's entirely workaroundable, technically this does break installs for X amount of time each time we update the repo index with a new package version. Long term, we should probably solve it.

Yes, exactly this - browser is caching

The http://repo.fpv.wtf/entware-armv7sf-k3.2/Packages.gz (+ similar) files are loaded by opkg update being run on device; it is proxied through the browser and the browser caches it - see here - it's been loaded from disk cache, cache headers specify 10 minutes; verified it persists through a reload.

You can't control/configure this via github pages as far as I can see - so options:

Override the headers - could do it with cloudflare? Or some other reverse proxy.... do we have servers/hosting anywhere ?
We could cachebust in the proxy - https://github.com/fpv-wtf/wtfos-configurator/blob/c46a8d6aadd8166b5703910f210696c268669531/src/utils/Proxy.js#L77 - downside to this is it will break all caching... not sure there is a lot else to break though?
Simpler / more correct than cachebusting; request has a cache: no-store parameter - https://developer.mozilla.org/en-US/docs/Web/API/Request/cache - we should use that.

I also have some concern looking at the http headers that there is a varnish cache on github; which would render the above pointless....

stylesuxx commented 1 year ago

cache: no-store parameter sounds good.

Regarding varnish cache - surely they invalidate the cache after gh-pages has been re-deployed, no? This would be interesting to validate, if it's not invalidated we would need to go with external hosting anyway.

Not sure if we have hosting at the moment, I am sure we could get some though..

benlumley commented 1 year ago

Tested the varnish cache thing; on push to gh-pages; it takes about a minute to roll out - so think we are safe on that front.

I'll do the cache: no-store thing later when i have goggles to test against - assuming you don't beat me to it!

j005u commented 1 year ago

Option c) could've been we always make sure the repo (but not the index) also contains any previous .ipk-s that got updated with the deploy at hand. But cache: no-store seems like the correct solution here.

We don't currently have any paid hosting. I'd say if we ever need it then we get an OC virtual card to pay for an AWS account for the team, so that at that point we can just spin things up relatively arbitarily as they're needed.

fpv-wtf / wtfos-configurator