search

grafana / google-bigquery-datasource

Google BigQuery Datasource Plugin for Grafana.
Apache License 2.0
25 stars 13 forks source link

Feature: Limit billable bytes per query #254

Closed duncan771 closed 4 months ago

duncan771 commented 4 months ago

What this PR does / why we need it: As an admin, I may want to control the cost of my query spend. With the BigQuery client this can be done by specifying the MaxBytesBilled on the QueryConfig.

This is a split from this PR by @abannachGrafana.

Description

This PR adds an configuration field to the datasource:

Screenshot 2024-05-08 at 2 46 06 PM Screenshot 2024-05-08 at 2 47 20 PM Screenshot 2024-05-08 at 2 47 37 PM Screenshot 2024-05-08 at 2 48 03 PM
duncan771 commented 4 months ago

@gabor Do you mind taking a look at this? It's the first half of Adam's initial PR that covers the MaxBillableBytes.

gabor commented 4 months ago

hi @duncan771 , thanks, i'm looking at the code.. the diff seems fine in general, i just have one question:

in google:

in our code:

i think it would be better to use the same phrases as Google uses.. WDYT?

duncan771 commented 4 months ago

hi @duncan771 , thanks, i'm looking at the code.. the diff seems fine in general, i just have one question:

in google:

in our code:

  • on the config page it is called Max billable bytes (also in the explanation text below)
  • in the soure-code it's name is MaxBillableBytes

i think it would be better to use the same phrases as Google uses.. WDYT?

@gabor I don't have a strong opinion one way or the other. I believe when we were writing this we used "Max Billable Bytes" as that felt like it was a little more clear to the end-user what we were intending for this to limit. Really just a semantics issue of Max Bytes Billed being something that's already been billed, whereas Max Billable Bytes would refer to a present or future-tense amount that we were limiting.

However, happy to keep it consistent with Google to keep it easier for downstream users to find Google's resources on it if you think that's best.

gabor commented 4 months ago

@duncan771 i see your point, max-billable-bytes is easier to understand. with that said i feel that making easier for the user to find google's resources is more important, so i recommend changing it, thanks!

duncan771 commented 4 months ago

@duncan771 i see your point, max-billable-bytes is easier to understand. with that said i feel that making easier for the user to find google's resources is more important, so i recommend changing it, thanks!

@gabor Sounds good! Just made the changes. Also, to help make sure this is better understood by any downstream user, we've linked to Google's relevant resources in the description under the Additional Settings.

gabor commented 4 months ago

thanks @duncan771 for the changes... i tried this out, and i have problems triggering the "error"... here is what i did:

  1. i have set the limit to 7
  2. i ran a query where the hint says This query will process 69 B when run.
  3. i ran the query
  4. i got results

i assumed i will get an error message saying this is over the limit.

are you able to trigger the error message?

duncan771 commented 4 months ago

thanks @duncan771 for the changes... i tried this out, and i have problems triggering the "error"... here is what i did:

  1. i have set the limit to 7
  2. i ran a query where the hint says This query will process 69 B when run.
  3. i ran the query
  4. i got results

i assumed i will get an error message saying this is over the limit.

are you able to trigger the error message?

@gabor I've been able to trigger the error message. I posted some photos above of just that, but I haven't tried it with such a small query - just a small value for MaxBilledByets. I just tested it again with a limit of 7 bytes and a query as small as 130 B and it still failed. Do you mind sharing your query to see if there's something unique about it causing it to run?

gabor commented 4 months ago

@duncan771 thanks, i chose a larger dataset, and now i was able to reproduce the error message 👍