alan-turing-institute / uatk-spc

Synthetic Population Catalyst
https://alan-turing-institute.github.io/uatk-spc/
MIT License
20 stars 12 forks source link

SPC v2 #48

Closed dabreegster closed 1 year ago

dabreegster commented 1 year ago

This is the big new version, with these changes (please edit where I'm wrong):

This PR has changes to the R scripts, Rust pipeline code, and web app.

Release-blocking tasks:

There are other ideas we might explore later, like squeezing down output file size by lazily calculating person-diary matches.

dabreegster commented 1 year ago

I reran everywhere. 294 successes, 96 failures. 124GB output uncompressed. Currently compressing stuff and then will upload to v2 directory.

Failed regions were all due to missing OA data, as expected. Scotland, most of the config/special regions.

South Yorkshire in 2022 failed due to a partial download; I'll rerun that one and upload.

dabreegster commented 1 year ago

43GB gzipped, slowly creeping its way to Azure!

dabreegster commented 1 year ago

I'm ready to merge this into main. We can release v2.1 and smaller follow-ups to add in missing areas, and incrementally improve the web app and any docs. Awaiting @HSalat approval.