search

a-scie / ptex

Ship gouged out scies that repair their base on 1st run by fetching missing files.
Apache License 2.0
9 stars 2 forks source link

Enable some kind of caching for large URLs #96

Closed thejcannon closed 1 year ago

thejcannon commented 1 year ago

Now that scie-pants is consuming a per-platform PEX, some of those can be sizeable (>40MB). scie-pants itself caches the result of the operation of using the PEX, but I think it'd be beneficial to have ptex provide a layer of caching.

No hard requirement on how this is implemented.

One way might be some kind of caching via ETag, and issuing a HEAD and try to match the resulting ETag to the DB. Or have the user specify if it wants to assume the bytes are immutable for a given URL.

jsirois commented 1 year ago

Ptex is meant to be a static curl binary and pretty much nothing more (it does have the 1 JSON mapping affordance scie-jump relies upon). The scie-jump should be handling caching correctly and only downloading a lazy PEX file once when its declared in the lift manifest as such. If you're using ptex directly outside the scie-jump as a curl substitute, that means you're bootstrapped and should be able to implement your own fetching and caching in whatever runtime the scie props up for you. Unless you can provide more detail on why Ptex should expand its role to provide these extra services, I'm inclined to continue to keep it tight and focused and not do this.