H-Plus-Time
commented
3 months ago An additional ClientOptions, exposed on the object store itself and the Wasm wrapper, is needed here.
Actually determining at runtime whether a HEAD is required (only in situations where the size of the file actually matters (cases where it doesn't matter: formats like parquet essentially describe themselves once you have the last 8 bytes))... that can be somewhat tricky if all you have is a range and a size that might be the size of the file or just the range.
This is one area where this repo will have to diverge from upstream (probably permanently) - each of the stores authored here will need an extra method (get_range_conditional_head, maybe get_ranges_conditional_head) that does:
- internal client::get_opts, with the (very simple) wrapping logic present in the ObjectStore trait.
- Based on inspection of the result of get_opts from the internal client (which will need a marker of some sort for internally indicating that the reported sizes are inaccurate), dispatch a second get_opts with a HEAD request (or, if it isn't subject to the same restrictions, a GET zero-length). The results of that second call should be returned alongside the first, with the overall file size extracted from the HEAD and some sort of clear indicator to avoid relying on subsequent calls to non-conditional-head methods.
kylebarron
Overall this is fairly straightforward - replicate the set of fixes, additional structs (ContentRange) and some of the error-reporting states.
There is, however, one important deviation when running inside a browser: 'Content-Range' is not a CORS-safelisted property :grimacing: : https://developer.mozilla.org/en-US/docs/Glossary/CORS-safelisted_response_header .
The upstream behaviour treats this as an error (rightly, given the HTTP spec mandates support for the
Rangeheader -> support for theContent-Rangeheader, and CORS is inapplicable outside the browser) - in the browser case, this is probably a no-go, given the paucity of sites that include it inAccess-Control-Expose-Headers.Since the
meta.sizefield is not an Option type, it must be set to something - the only problem with that is the natural fallback value (what object store used to do, namely use the Content-Length field) is likely to be wildly inaccurate.