search

PiechZ / greybox_wrapped

greybox wrapped is a small project to display statistics about academic debates.
MIT License
4 stars 0 forks source link

Merge, or otherwise combine, with greybox-mysql2duck #85

Closed shippy closed 1 month ago

shippy commented 1 year ago

greybox-mysql2duck is where the data that this repository depends on originates. It also has two natural entrypoints for the combination:

  1. Meltano can actually run the dbt transformation as part of the MySQL conversion pipeline, thereby giving us a ready DuckDB database in data/ anytime we drop in a new SQL file there,
  2. Both repositories are containerized, so the docker-compose files could be combined.

Uniting the repositories allows us to (a) centralize the work and (b) think about it cohesively. Then again, they do work separately right now, so "don't fix what ain't broke" applies, and uniting the two would nest dbt far deeper into the file structure.

shippy commented 2 months ago

The feature/dbt branch on the other repo has the integration.. I had to make some subtle changes to the dbt configuration, and the whole thing now runs in a Docker container.

I think I should port it here and just change the Makefile commands to execute the dbt build. The whole thing might work outside of Docker, too, but that might get tricky.