crawler that replicates our ETL catalog into local DuckDB
API that can serve data and metadata when given variable id
A lot could be refactored, but it is in a shape to serve some basic requests. It was only tested on handful of datasets (primarily dataset 941), but I'm adding more. It is currently one big commit with all the parts inside because everything's changing quickly and I've rewritten its structure a few times already. I can split it into smaller PRs (or commits), although it'd be great to get some feedback first to avoid having to rewrite it again.
Initial API prototype with two components:
A lot could be refactored, but it is in a shape to serve some basic requests. It was only tested on handful of datasets (primarily dataset 941), but I'm adding more. It is currently one big commit with all the parts inside because everything's changing quickly and I've rewritten its structure a few times already. I can split it into smaller PRs (or commits), although it'd be great to get some feedback first to avoid having to rewrite it again.
Limitations