bendnorman
commented
6 months ago - [ ] Consolidate package management. Remove package specified in requirements.txt and environment.yml and specify everything in pyproject.toml.
- [ ] set up terraform
- [ ] Rip out PUDL dependency
- [x] Setup CI that runs unit and integration tests: In Progress: #309
- [ ] Setup dependabot so packages stay up to date. Requires ^
- [x] Setup CD that updates big query when code is pushed to dev or main: In Progress: #309
- [ ] pre-commit CI!
- [ ] Setup conda lock file automation. Could be helpful for managing non python dependencies within docker container
- [ ] Gitbook automation in CI
- [ ] iron out how we archive the data! we have some things in git-lfs, some things in gs://dgm-archive and somethings in gridstatus-archive. I think everything should be stored in GS. How do we want to version the data? What are all of our datasets and how often are they updated?
- [ ] Use sqlite or duckdb instead of postgres: Main benefit of this is to remove docker and make the repo setup more similar to PUDL and we can remove Docker which simplifies things.
- [ ] Adopt dagster