search

cms-dpoa / cloud-processing

Exploring the usage of public cloud resources for CMS open data processing
GNU General Public License v3.0
0 stars 0 forks source link

Problem with accessing billing information from CLI #17

Closed tomcordruw closed 7 hours ago

tomcordruw commented 5 days ago

Because it would be useful to also get a breakdown of costs and billing information, I tried to see whether it's possible to monitor them from the terminal instead of the google cloud console browser interface.

After activating the API I tried getting billing information via the CLI as following:

gcloud billing budgets describe <project-name> --billing-account=<billing-account-id>

But I wasn't able to resolve this resulting error:

ERROR: (gcloud.billing.budgets.describe) PERMISSION_DENIED: The caller does not have permission. This command is authenticated as <email-address> which is the active account specified by the [core/account] property

I attempted adding roles with the correct permissions (Billing Account Viewer) but wasn't able to find the role and adding it as a custom role said that it wasn't possible to add these permissions via custom roles. I also tried re-authenticating my account with gcloud CLI.

katilp commented 1 day ago

OK, sorry for the delay, that's something I need to activate then. I'll do this.

katilp commented 1 day ago

Hmm, I get the same. As we are both owners I believe we should have those rights included.

However, I'm not sure if the budget gives any additional information about the breakdown of the costs.

I see it in use only when setting budget limits. Do you know better?

katilp commented 1 day ago

https://www.googlecloudcommunity.com/gc/Google-Cloud-s-operations-suite/billing-report-through-cli/m-p/595959

katilp commented 1 day ago

@tomcordruw if you like you can check if this brings any values compared to the simple GCP Web UI billing view. The Web UI granularity is 24h and it can be annoying (e.g. when timing two different things on the same cluster)

BigQuery brings an additional cost but I think it is minimal.

subash-taranga commented 1 day ago

yes i am going through BigQuery

katilp commented 1 day ago

@subash-taranga I do not think both of you must inspect that. I suggest that @tomcordruw has a look first and provides a summary of instructions for us to use in the evaluation. Meanwhile, you could start evaluating the cost of and time of downloading a certain amount of data from a bucket and an nfs disk. I will open a separate issue for that.

tomcordruw commented 1 day ago

@subash-taranga I do not think both of you must inspect that. I suggest that @tomcordruw has a look first and provides a summary of instructions for us to use in the evaluation. Meanwhile, you could start evaluating the cost of and time of downloading a certain amount of data from a bucket and an nfs disk. I will open a separate issue for that.

That would be helpful, let's do it like that.

subash-taranga commented 1 day ago

Sure i will start looking at new assignment from now which is https://github.com/cms-dpoa/cloud-processing/issues/19

tomcordruw commented 7 hours ago

Now that we have decided to use BigQuery instead for the cost analysis and since reading the billing information from CLI doesn't seem to be an implemented feature yet, this issue can be closed.