Figure out the permissions required to run the GCP Billing integration

[zmoog]

I want to update the GCP Billing Metrics integration docs page with a details about permissions.

My plan is to set up a service account with no permissions and add one at a time.

[zmoog]

Create a Service Account

• Visit IAM & Admin > Service Accounts
• Click on Create Service Account
• Set "mbranca-billing-metrics" as Service account name
• Click on Create and continue
• Click on Done

I don't want to assign any role for now.

Add a key

Select the new Service Account we just created and

• Select the Keys tab
• Click on Add Key > Create new key with the type JSON
• Click on the Create button

Download and store the .json file in a safe place.

[zmoog]

Set up GCP configuration file

I am going to use Metricbeat in debug mode. I enable the GCP module with the following configuration:

- module: gcp
  metricsets:
    - billing
  period: 24h
  project_id: <my project id>
  credentials_json: <my credentials json string>
  dataset_id: "billing_testing"
  table_pattern: "gcp_billing_export_v1"
  cost_type: "regular"

Optional, but nice. Format the credentials JSON string with jq before setting it in the credentials_json configuration option:

cat ~/Downloads/<project>-f4bf474898e4.json | jq -c

[zmoog]

Next step is starting GoLand in debug mode following the instructions at #28.

Here we are:

[zmoog]

API: Datasets

The first API call is Datasets() to list the existing datasets.

After executing the following piece of code:

    for {
        dataset, err := dit.Next()
        if errors.Is(err, iterator.Done) {
            break
        }

We don't get a missing permission error, but just a "no more items in iterator" error. No more items is one of the possible conditions, so the the executions stops.

The missing permissions condition do not show up in this interaction. The metricset does not log anything. This makes troubleshooting much more difficult.

[zmoog]

I am making a small change by calling the PageInfo() function on the iterator to check how many datasets exist:

    pageInfo := dit.PageInfo()
    if pageInfo.Remaining() == 0 {
        return tables, fmt.Errorf("no datasets found in project %s (check your project ID and the permissions of your service account.)", client.Project())
    }

With no permission, the metricset logs the errors returns.

[zmoog]

Okay, now let's assign a role to this service account to see if we get some actual datasets from GCP.

I'm assigning the With the BigQuery Data Viewer role that has the following permissions:

bigquery.datasets.get
bigquery.datasets.getIamPolicy

[zmoog]

To assign the "BigQuery Data Viewer" to the service account you can:

- IAM & Admin > IAM > View by role
- Click o **Grant access**

The add:

• the service account as Principal
• the "With the BigQuery Data Viewer" as role

[zmoog]

If I run Metricbeat again, the metricset is able to list datasets and tables:

[zmoog]

We need to add a log message to avoid the metricset to silently fail when it does not have the permission to list datasets and tables.

Something simple like the following would do:

if len(tableMetas) == 0 {
    m.logger.Errorf("no tables found in dataset %s with pattern %s; check your settings and see if the service account has permission to list datasets", m.config.DatasetID, m.config.TablePattern)
    return nil
}

This statement generates a log message like this one:

{
  "log.level": "error",
  "@timestamp": "2023-10-06T09:32:25.318+0200",
  "log.logger": "billing",
  "log.origin": {
    "file.name": "billing/billing.go",
    "file.line": 148
  },
  "message": "no tables found in dataset billing_testing with pattern gcp_billing_export_v1; check your settings and see if the service account has permission to list datasets",
  "service.name": "metricbeat",
  "ecs.version": "1.6.0"
}