search

evidence-dev / evidence

Business intelligence as code: build fast, interactive data visualizations in SQL and markdown
https://evidence.dev
MIT License
4.5k stars 216 forks source link

Optimize `run sources` #2593

Open mcrascal opened 1 month ago

mcrascal commented 1 month ago

Improve the speed of running sources and writing parquet, improve logging on source runs, reduce memory footprint if possible.

archiewood commented 1 month ago

perhaps put logging behind debug flag

ud3sh commented 1 month ago

Think it would be really helpful if npm run sources logs includes under each source query 1) the total time of the query exec, and 2) and where possible depending on the connector the query exec time from db stats (possibly in debug mode only)

This would help with optimizing source queries, as well as understand where there are bottlenecks during deployments/builds

ud3sh commented 1 month ago

Think it would be really helpful if npm run sources logs includes under each source query 1) the total time of the query exec, and 2) and where possible depending on the connector the query exec time from db stats (possibly in debug mode only)

This would help with optimizing source queries, as well as understand where there are bottlenecks during deployments/builds

Archie opened an issue for this https://github.com/evidence-dev/evidence/issues/2629 //cc @csjh

hughess commented 1 month ago

Adding a note here for better error handling when query result is too large, or region is not set correctly (bigquery example)

https://evidencedev.slack.com/archives/C023LRB9Z40/p1727121470531939?thread_ts=1727101387.250899&cid=C023LRB9Z40