Authentication not working as expected when running GKE with a service account

[psalaberria002]

Hello,

I have a GKE cluster configured to use a service account for the nodes. In addition I have created a separate service account with elevated permissions for the GCE Jenkins plugin.

I noticed that the GCE Jenkins plugin ignores the Service Account Credentials provided in the configuration section, and instead authenticates with the service account that the nodes are running as.

I get the following error: Could not list regions in project <project>. Once given extra permissions to the first service account, everything works.

This is unexpected and very confusing. Hopefully it's an easy fix.

Thanks.

[stephenashank]

Hi @psalaberria002, can you tell me whether the first service account (on the cluster) had the Service Account User role before you added more permissions? If not then this was likely the missing permission, as the cluster service account must be authorized to act as the second service account using the selected credentials.

If it already had those permissions, are these service account credentials for the same project, and if so are they both stored in Jenkins? If that is case, this might be related to https://github.com/jenkinsci/google-oauth-plugin/issues/12. Recently, I found a workaround which is that the Project ID value configured in the UI when creating the credentials does not need to be a Project ID, it can be anything, allowing you to use unique identifiers for each key, and select the correct credential from the drop down.

If neither of those were the issue then this is likely a bug in the client factory, and I might need more info from you to pinpoint the problem.

[victorboissiere]

Hi, I have the same issue:

However, I use only one service account and do not need a second one. I've checked access scope from the compute instance (full API access). I've also uploaded the key on the instance and tested manually with gcloud compute regions list and it works without issues. I do not understand what is the issue and where to look. Nothing shows up in Jenkins logs.

[victorboissiere]

Ok on my side I've rollbacked Jenkins to 2.211 (from 2.212) and the issue is fixed!

[johannespostler]

I'm experiencing the same issue with only one service account on Jenkins 2.204.1 and plugin version 4.2.0 The service account has all permission mentioned in the docs and a google example.

Any ideas?

[isarkis]

We are experiencing similar issues - getting 'Could not list regions in project dropcam-hq'. We also see an error in the jenkins log:

2020-03-13 01:00:26.342+0000 [id=679] SEVERE c.g.j.p.c.AutofilledNetworkConfiguration$DescriptorImpl#doFillNetworkItems: Error retrieving networks com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden { "code" : 403, "errors" : [ { "domain" : "global", "message" : "Required 'compute.networks.list' permission for 'projects/**'", "reason" : "forbidden" } ], "message" : "Required 'compute.networks.list' permission for 'projects/***'" }

However, service account in question does have the following permissions:

roles/compute.instanceAdmin roles/compute.networkAdmin roles/editor roles/iam.serviceAccountUser

[Mukhtarali212]

Hi , I have the same issue "Could not list regions in project" in the jenkins and configure correctly follow the google cloud documentation. and configure the credentials but i don't know where i am going wrong.

please suggest me the right one, Thanks

[psalaberria002]

We figured it out after a couple of hours of debugging.

There are two types of credentials in the Google Oauth Credentials plugin. One is based on a key, the other one based on instance metadata.

When creating new credentials of any of these types, the only input field is the Google project name, which is then used as the ID for the credential. The problem comes when a new credential is added for the same project. The old credentials are still available to Jenkins but hidden in the UI.

When selecting the credential in a dropdown, you may think you are selecting the latest one, but that's not happening. The oldest credential with that ID is used instead.

After deleting the credentials (we had around 5 with the same ID :)) and adding the new one with the right permissions it started to work.

I will leave a Terraform Snippet for reference:

# Service Account for creating Windows agents in GCP
resource "google_service_account" "jenkins_windows_agent_creator" {
  project      = local.project
  account_id   = "jenkins-windows-agent-creator"
  display_name = "Jenkins Windows Agent Creator"
}

resource "google_project_iam_member" "jenkins_windows_agent_creator_compute_instance_admin" {
  project = local.project
  role    = "roles/compute.instanceAdmin.v1"
  member  = "serviceAccount:${google_service_account.jenkins_windows_agent_creator.email}"
}

resource "google_project_iam_member" "jenkins_windows_agent_creator_compute_network_viewer" {
  project = local.project
  role    = "roles/compute.networkViewer"
  member  = "serviceAccount:${google_service_account.jenkins_windows_agent_creator.email}"
}

# Service Account used for running Windows agents in GCP
# It does not need any access to other Google APIs
# Relying on the default compute SA of the project is considered bad practice,
# it's better to create an SA with limited or no permissions at all.
resource "google_service_account" "jenkins_windows_agent_runner" {
  project      = local.project
  account_id   = "jenkins-windows-agent-runner"
  display_name = "Jenkins Windows Agent Runner"
}

# Allows the Creator to spin-up VMs that run as the Runner
resource "google_service_account_iam_member" "jenkins_windows_agent_sa_user" {
  role               = "roles/iam.serviceAccountUser"
  member             = "serviceAccount:${google_service_account.jenkins_windows_agent_creator.email}"
  service_account_id = google_service_account.jenkins_windows_agent_runner.name
}

The SA that runs the Jenkins VM does not need any permissions.