Open Bonkles opened 1 month ago
Two pieces of information would be super useful in the manifests:
serialized_size
value of each file's FileMetaData - geoarrow-wasm can (with a minor tweak) forward that through to the with_footer_size_hint
method of each file's reader instance. Effectively cuts 1 of the 4 requests (instant disk cache hit).It likely won't make a huge difference to this repo given the bounding box optimizations (and the CF distro of course), but a 50% cut in metadata requests is still nice (that and I reckon a bit of offline behaviour/speculative read-ahead is possible with that). Also assuming these manifests make their way out to general use, broad bounding boxes + non-spatial filters will benefit greatly.
Agreed @H-Plus-Time - the manifests are supposed to be general purpose info that helps everybody, so this is great info for me to consider. While I have this site in mind as a use case, I want to make sure useful info goes in for others.
This PR attempts to improve our download times by reducing file count in two ways: 1) The use of pre-calculated manifests with bboxes per parquet file 2) omitting any non-visible themes from the download data.
The first item is accomplished from the manifests generated by @Bonkles/manifest-generator to speed up and improve our download experience.
These two bits of logic allow us to make refinements to the total catalog of files we might need to consider- greatly decreasing the number of files to consult.
So, when the user clicks the download button, we can massively pare down the # of HTTP requests/data loading required to assemble a valid catalog. If the user wants buildings, we'd previously have to send 4 HTTP requests per file, a total of about 800+ requests, before we could start downloading data.
Still to-do:
This PR is in draft form as I'm waiting for the new geoparquet reader code revamp to land, then I can really test.