Question : is it possible to run a pipeline only when a source table has changed (BigQuery)

[djouallah]

I start using Dataform yesterday, although I am not a dev, I was up and running in less then half an hour, great work, I am using BigQuery, my data change randomly, maybe once a day, week, month !!!, is there a way to run the pipeline only when one of source tables change ?

[BenBirt]

Hi @djouallah ! Glad to hear you got it all set up, thanks for the feedback!

Currently we have the following answers to running pipelines on a non-scheduled basis:

• Use the Runs API, triggered whenever you want: https://docs.dataform.co/dataform-web/using-the-api
• Use run caching. Note that this feature is still in alpha. Run caching will allow you to configure your pipeline to run as often as you need, but will only actually run the parts of your pipeline that need to re-run because something changed: https://docs.dataform.co/guides/configuration#enable-run-caching-to-cut-warehouse-costs

Note that we don't always keep an eye on GitHub issues, so you'll usually get quicker answers to your questions on our Slack - please join us there! https://dataform.co/slack