GCP metricset does not collect Google Cloud Org ID and display name

[zmoog]

It seems the compute metricset intentionally sets (1) both cloud.account.name and cloud.account.id (2) with the value of the project_id resource label (3):

Users expectation is to have:

• the project ID in the cloud.project.id field (currently missing)
• the Google Cloud ORG ID in the cloud.account.id field (currently contains the project ID)
• the Google Cloud ORG display name in the cloud.account.name field (currently contains the project ID)

These expectations are in line with the ECS semantics:

• cloud.project.id: schemas/cloud.yml#L133-L151
• cloud.account.id and cloud.account.name: schemas/cloud.yml#L98-L119

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[lalit-satapathy]

Assigned @Linu-Elias

[zmoog]

Here is the reference to the cloud fields in ECS to double-check that the cloud.* field semantics are in line with ECS.

https://www.elastic.co/guide/en/ecs/current/ecs-cloud.html

[gpop63]

@zmoog IIRC getting organization info requires an additional API call, right?

We could use resourcemanager.GetProject which returns a Project struct. This struct has a field Parent that can contain organization info:

// Optional. A reference to a parent Resource. eg., organizations/123 or // folders/876.

[zmoog]

IIRC getting organization info requires an additional API call, right?

Yep! Unfortunately, I didn't keep this info about my research when I created the issue, but I remember the same info.

So I guess we need an additional API call to collect the organization info for each unique project.

[endorama]

A couple of notes :)

GCP resources are organized in a hierarchy where at the root there is the organization. resourcemanager.GetProject returns a Parent but to reach the root organization you would need to traverse up the tree up to the root node with multiple API calls. Would be better to use the getAncestry API if possible, ProjectService.GetAncestry in the Golang SDK, which returns a GetAncestryResponse that contains a list of parents. The last is the root/org.

Projects may not be part of an organization (as mentioned by "Create a project" docs, is possible to have the "No organization" value when selecting project location). I'm not sure how this works in detail because there is no clear documentation about it. I've seen it related to managed and non-managed Google accounts, but in general is considered a bad practice because of multiple limitations (especially around IAM management). Still is a possible corner case that must be handled.

[zmoog]

We are lucky, then! It seems that at least our test project falls into this "no organization" category, so we can double-check how it works.

[Linu-Elias]

Thanks @endorama, we are now using ProjectsService.GetAncestry in which we get the root/org ID and get the particular organization details (specifically aiming to collect organization name) using- OrganizationsService.Get

As we now know that projects may not be part of an organization, and all our projects fall into "no organization" category so OrganizationsService.Get doesn't really return much. We would need projects which are a part on an actual organization for testing purpose.

[zmoog]

@Linu-Elias I have a Google account with Cloud Identity Free which allows me to have an actual Google Cloud organization. We can start by using this for testing.

[Linu-Elias]

@zmoog, could you please share the details of the Google account? That would be helpful. If you have an existing project that belongs to an organization, sharing the credentials of a service account with the necessary permissions would also work.

[zmoog]

could you please share the details of the Google account? That would be helpful. If you have an existing project that belongs to an organization, sharing the credentials of a service account with the necessary permissions would also work.

Yep! The quickest option is to share a service account. Let me set up one for you.

[zmoog]

Hey @Linu-Elias, I set up the following structure, and there is a service account with access to the "My First Project" project.

I shared the service account details using 1Password. See the DM for the link.

In the project, there is a VM running and I can collect compute metrics using the following gcp module config:

- module: gcp
  metricsets:
    - compute
  region: "europe-west12"
  project_id: "<redacted>"
  credentials_json: '<redacted>'
  exclude_labels: false
  period: 1m