search

opensource-observer / oso

Measuring the impact of open source software
https://opensource.observer
Apache License 2.0
69 stars 16 forks source link

Try Dagster embedded-elt for database replication #1318

Closed ryscheng closed 3 months ago

ryscheng commented 5 months ago

What is it?

We've been having all sorts of issues with the Airbyte connector. See this issue for a summary https://github.com/opensource-observer/oso/issues/1229

Dagster seems to have built-in support for ELT using Sling https://docs.dagster.io/integrations/embedded-elt

We can try this instead to see if that can auto-magically work with fewer modifications

ryscheng commented 5 months ago

Here's the Sling documentation for replication https://docs.slingdata.io/sling-cli/run/examples/database-to-database

ryscheng commented 5 months ago

Apparently we can choose between dlt or Sling

https://docs.dagster.io/integrations/embedded-elt/dlt

According to @ravenac95 they have better docs on how to write a connector

ryscheng commented 5 months ago

Another possibility is CloudQuery

https://github.com/cloudquery/cq_dagster_embedded_elt

ryscheng commented 3 months ago

https://github.com/opensource-observer/oso/pull/1733 Done