Deterministic Database - Githubissues

EFREI-2026-Mastercamp-IT-1 / Backend

Other

0 stars 0 forks source link

Deterministic Database #31

Closed Malix-off closed 1 month ago

Malix-off commented 1 month ago

Context

big dataset file size (~700MB)
immutable dataset
mutable database schema
5 contributors

Solution

Instead of using a statically updated .db file, deterministically build the database from the raw files (data/raw/*.csv)

osbyrne commented 1 month ago

Closed as Not planned

Reason

Dataset drastically reduced size (dropped some data useless for our current use-case)
Compressing SQLite is enough (but still cannot be asynchronously worked on without conflicts)

Workaround

Using the SQLite database directly (maybe compressing it beforehand)