search

kestra-io / plugin-gcp

Apache License 2.0
9 stars 10 forks source link

feat(metrics): add BQ total return rows in metrics #316

Closed aurelienWls closed 8 months ago

aurelienWls commented 10 months ago

What changes are being made and why?

Add in the Kestra metrics the number of lines return by a BigQuery query.

This is usefull to track overtime the number of lines return by a BigQuery.Query task thanks to Kestra metrics dashboard.

This change involved loading all the data return by the query in the metrics method. This could lead to performance issue in case the query return a huge amount of data. This change has not been tested with huge query.

How the changes have been QAed?

Tested locally

Setup Instructions

loicmathieu commented 8 months ago

There is already a total.rows metric that return the same metric value (queryJob.getQueryResults().getTotalRows() so it seems to be already covered.

This metric will not be displayed if neither store, fetch, or fetchOne is true as in this case the results are not fetched from BQ.