search

dagster-io / hooli-data-eng-pipelines

Example Dagster Cloud code for the Hooli Data Engineering organization.
75 stars 15 forks source link

Lopp add more frequency #44

Closed slopp closed 1 year ago

slopp commented 1 year ago

This PR adds another code location to hooli that contains a project that will serve two purposes:

  1. Adds an example of DynamicOut which is helpful when showing off Dagster's ability to run tasks in parallel
  2. Adds a schedule to run ^ with more frequency. This will make the overview page more realistic. This schedule specifically avoids Snowflake so that the example project is self-contained and increasing the run frequency doesn't impact our snowflake demo costs.

While I am generally hesitant to add yet-another-project to Hooli, I think this addition is worthwhile. The main tradeoff is that in demos we'll want to be careful to filter on just the data_eng_prod code location to keep the asset graph "clean". I also considered just adding a frequently running job to the Hooli basics code location, but I hesitated to do that in order to keep that code location truly minimal and basic.

github-actions[bot] commented 1 year ago

Your pull request is automatically being deployed to Dagster Cloud.

Location Status Link Updated
batch_enrichment View in Cloud Oct 18, 2023 at 06:20 PM (UTC)
data-eng-pipeline View in Cloud Oct 18, 2023 at 06:20 PM (UTC)
basics View in Cloud Oct 18, 2023 at 06:20 PM (UTC)