Initially, this project used Amazon's RDS as a database backend. However, this proved to be costly and not very practical.
To make flatsurvey easier to use, the database should just live in local files instead. We do not need concurrent access here and the database is, for the foreseeable future, not going to grow beyond what can be easily stored in RAM.
[ ] Move the database to YAML files in a flatsurvey-data repository. Each kind of object should live in a separate directory and each survey run should live in a separate .yaml file, e.g., ngon/2022-05-05.yaml.
[ ] Move the pickles to a flatsurvey-data-pickles repository. (We can directly download from there as we can with S3 but we can also download the entire set of pickles to a local disk and then get much faster access to bulk convert the database.)
[ ] Rewrite the pickles to use a more efficiient protocol. Currently, the vectors (directions we used to compute flow decompositions) use 8GB on disk. Ideally, but probably not realistic, would be a (compressed) format that is compatible with actual Python pickles.
Initially, this project used Amazon's RDS as a database backend. However, this proved to be costly and not very practical.
To make flatsurvey easier to use, the database should just live in local files instead. We do not need concurrent access here and the database is, for the foreseeable future, not going to grow beyond what can be easily stored in RAM.
ngon/2022-05-05.yaml.