Allow users to configure number of threads in warehouse connection settings

dataform-co / dataform-web-tracking

Dataform is a collaborative data modelling platform that enables analysts and engineers to manage complex data models in SQL

https://dataform.co

MIT License

2 stars 0 forks source link

Allow users to configure number of threads in warehouse connection settings #112

Closed lewish closed 5 years ago

lewish commented 5 years ago

The threads property of the Profile profile can't be overridden in dataform.co, and currently defaults to 16 for BigQuery: https://github.com/dataform-co/dataform/blob/master/api/dbadapters/bigquery.ts#L23

We should allow overriding this setting in the Database settings page, and probably set a sensible range, something like 1-32: https://github.com/dataform-co/dataform-co/blob/master/ts/app/database/index.tsx

Moved from dataform-co/dataform-co#187 /cc @lewish

lewish commented 5 years ago

@tanguyco for updates

BenBirt commented 5 years ago

We plan to support users to understand the critical runtime path of their pipelines in order to help them optimise runtime; this issue will be reopened at that time.