dagster-io / hooli-data-eng-pipelines

Example Dagster Cloud code for the Hooli Data Engineering organization.
69 stars 11 forks source link

Add Airbyte #8

Open slopp opened 1 year ago

slopp commented 1 year ago

Add an example that uses: https://hooli-airbyte.purina.elementl.dev/

One consideration with this example is deciding how to include it such that the repository still runs locally with minimal dependencies.

It may also be worth waiting until the airbyte-connections-as-code changes land.

airbyte-airbyte-webapp-svc.airbyte.svc.cluster.local

kubectl port-forward -n airbyte svc/airbyte-airbyte-webapp-svc 9000:80

Example starting point: https://github.com/dagster-io/dagster/tree/master/examples/assets_modern_data_stack

slopp commented 1 year ago

Depends on https://github.com/dagster-io/dagster/pull/10224