Enable compilation of lib/core crate to WASM

[sgreenbury]

• [x] Fix wasm specific build for polars
• [x] Fix wasm specific build for popgetter core
• [ ] Add wasm specific implementations in popgetter core (or potentially a popgetter-wasm). Features that may need to be updated:
  • [x] HTTP requests require a WASM client (e.g. https://crates.io/crates/gloo_net). reqwest v0.11 compiles to wasm, so implementation updated to use this.
    • [ ] Check if reqwest v0.12 is also compatible
  • [ ] Reading parquet efficiently using HTTP range request
  • [ ] Adding WASM-specific tests (e.g. testing the reading parquet)

[sgreenbury]

Polars compatibility

See https://github.com/Urban-Analytics-Technology-Platform/popgetter-cli/commit/46f04329beb5c99561070b26fc3f0e3842aa0bc9

• Not all off the feature flags currently used can be compiled to WASM, specifically "http" and "streaming" that bring in a dep on mio. To fix this we can remove these specific features when compiling to WASM by creating target specific dependency on polars through wasm and not_wasm features that optionally include these features.
• A required feature includes parquet that also includes multiple compression features with lz4 not compiling to WASM. A workaround is to point to a fork of polars with compression features included in the parquet feature that can compile to WASM.
  • [ ] consider alternative approaches that do not change the current parquet feature
  • [ ] consider solutions that do not introduce a git dep

Other compatibility

• Flatgeobuf dep is updated to 4.2.1 for WASM compatibility
• Reqwest 0.11 can compile to WASM
• Not all tokio features can be compiled to WASM, so subset required ("rt", "macros") are included

Revisions to popgetter

• [ ] Some required polars parquet functionality may not be functional without "http" and "streaming" features. For example, the scan_parquet call here appears to require "http"/"streaming". An alternative that is compatible with WASM features and ideally can efficiently only do an http request for the columns required from the metrics, the needs to be implemented and tested
• [ ] Getting the countries also used scan_parquet, so an alternative with reqwest for WASM is implemented but needs to be tested
• [ ] flatgeobuf functionality with BBox would be good to test with WASM

Next steps

• Consider/implement above identified tasks
• Add tests in popgetter lib/core to test functionality when compiled to wasm32-unknown-unknown with wasm features
• Explore integration with browser with open issues in popgetter-browser and begin implementation of the Rust backend that uses popgetter

[dabreegster]

In case it's helpful, https://crates.io/crates/streaming-http-range-client and https://github.com/michaelkirk/geomedea/tree/main/geomedea-wasm might be helpful to figure out how to do streaming HTTP range requests in WASM. And if you want to chat about this stuff, join the georust Discord https://discord.gg/Fp2aape and ping Michael and Kyle

[sgreenbury]

Thanks @dabreegster for the helpful suggestions! polars also has an implementation (though currently this cannot used directly with WASM since the "cloud" feature depends on a multi-threaded tokio runtime) that might be useful for understanding how to construct the required range requests and convert to a polars dataframe.

I've added a functional WASM implementation for getting the metrics so that it does not block popgetter-browser but this does not have a range request for subsets of columns/rows yet https://github.com/Urban-Analytics-Technology-Platform/popgetter-cli/pull/76

[sgreenbury]

Next steps from discussion @stuartlynn @sgreenbury @andrewphilipsmith:

• [ ] Update the dep on polars fork to the rev (not branch)
• [ ] Two options:
  • [ ] Use polars deps all in Rust. Explore deps that polars uses (arrow2, parquet implementations) and other options too (parquet-wasm) that could do the range request
  • [ ] Download from JS/TS:
  • 1. Return a list SQL queries. Return to JS/TS. (WASM flag) (@sgreenbury)
  • 2. Call duckdb with each SQL query (uses a SQL query as input) with this data to perform the range requests on the parquet files. See v0 popgetter browser (@stuartlynn)
  • 3. svelte-maplibre join
  • This would also enable additional displaying in JS/TS.
• [ ] Confirm flatgeobuf is using range requests
• [ ] For displaying in the web app, consider using pmtiles as used in the origianal web app version (more efficient)
  • [ ] 1. Manually upload to Azure to begin with (@stuartlynn)
  • [ ] 2. Revisit using the docker image for making the pmtiles and be in the popgetter data pipeline

[sgreenbury]

See 153 for an alternative approach without "parquet" feature.