snowplow-archive / dataform-data-models

Snowplow Incubator project for Dataform SQL data models for working with Snowplow data. Supports BigQuery only
Apache License 2.0
4 stars 0 forks source link

Add source project and target project to tables #8

Open awoehrl opened 3 years ago

awoehrl commented 3 years ago

Some Dataform projects use a more complex setup with:

  1. Different source and target GCP projects
  2. Serparate dev and prod environments (https://docs.dataform.co/dataform-web/scheduling/environments#example-use-separate-databases-for-development-and-production-data)

As the tables are hard-coded in this model right now, these setups don't work out of the box right now. A simple fix could be to introduce variables for source and target:

const input_project = "snowplow-raw-302115";
const output_project = dataform.projectConfig.defaultDatabase;

Also maybe this could be solved in the SQLX files itself by using the dataform config functions?

config {
  database : "snowplow-raw-302115",
  schema: "rt_pipeline_prod1",
  name: "events",
  description: "Raw Snowplow events table"
}