Fog::Storage::GoogleJSON::Real#iam_signer is using the wrong API scopes with Workload Identity

[peikk0]

I was investigating why running GitLab in Kubernetes with Workload Identity instead of a service account key resulted in ACCESS_TOKEN_SCOPE_INSUFFICIENT errors:

irb(main):045:0> credentials = {provider: 'Google', google_project: 'my-gitlab-project', google_application_default: true}
=> {:provider=>"Google", :google_project=>"my-gitlab-project", :google_application_default=>true}
irb(main):046:0> raw_config = {enabled: true, connection: credentials, remote_directory: 'my-gitlab-project-uploads', storage_options: {}, consolidated_settings: false}
=>
{:enabled=>true,
...
irb(main):047:0> config = ObjectStorage::Config.new(raw_config)
=>
#<ObjectStorage::Config:0x00007f3d35837538
...
irb(main):048:0> du = ObjectStorage::DirectUpload.new(config, 'tmp/foo/bar', has_length: true)
=>
#<ObjectStorage::DirectUpload:0x00007f3d35fdbc58
...
irb(main):049:0> du.get_url rescue puts $!.body
{
  "error": {
    "code": 403,
    "message": "Request had insufficient authentication scopes.",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT",
        "domain": "googleapis.com",
        "metadata": {
          "method": "google.iam.credentials.v1.IAMCredentials.SignBlob",
          "service": "iamcredentials.googleapis.com"
        }
      }
    ]
  }
}
=> nil

Digging into fog-google I have found 2 issues causing this:

1. Fog::Storage::GoogleJSON::Real#iam_signer is using the wrong API scope, it should be https://www.googleapis.com/auth/iam instead of https://www.googleapis.com/auth/devstorage.full_control: https://github.com/fog/fog-google/blob/30685949ff7688e1066e4c8625480caffc7a495f/lib/fog/storage/google_json.rb#L32
2. Fixing the API scope above alone is not enough, as Fog::Storage::GoogleJSON::Real#iam_signer seems to be re-using the same session and auth credentials as with the storage operations, I haven't been able to find where and why yet but changing GOOGLE_STORAGE_JSON_API_SCOPE_URLS to https://www.googleapis.com/auth/cloud-platform did indeed make it work, though that's not the ideal solution

[peikk0]

@Temikus as the last person to update that file, can I ask for your attention on this bug?

[stanhu]

@peikk0 https://github.com/fog/fog-google/pull/629 fixes this problem. In more detail:

1. initialize_google_client, sets the default authorization with GOOGLE_STORAGE_JSON_API_SCOPE_URLS. This causes all requests by default to use that scope: https://github.com/fog/fog-google/blob/afd289d97890ae6f30cc671e746a0b96808cd170/lib/fog/google/shared.rb#L77-L82
2. In addition, this code doesn't actually do anything: https://github.com/fog/fog-google/blob/afd289d97890ae6f30cc671e746a0b96808cd170/lib/fog/storage/google_json/real.rb#L19-L22

If we look at apply_client_options: https://github.com/fog/fog-google/blob/afd289d97890ae6f30cc671e746a0b96808cd170/lib/fog/google/shared.rb#L92-L98

For @iam_service, options only contains the google_api_scope_url key, but the code looks for google_client_options. Even if google_client_options were present, google_api_scope_url would never be used since client_options doesn't set the scope:

irb(main):020:0> iam_service = ::Google::Apis::IamcredentialsV1::IAMCredentialsService.new
=>
#<Google::Apis::IamcredentialsV1::IAMCredentialsService:0x00007a63fa2ef080
...
irb(main):021:0> iam_service.client_options.members
=> [:application_name, :application_version, :proxy_url, :open_timeout_sec, :read_timeout_sec, :send_timeout_sec, :log_http_requests, :transparent_gzip_decompression]

As described in https://github.com/googleapis/google-api-ruby-client/blob/main/docs/usage-guide.md#passing-authorization-to-requests, we need to request a new access token with the right scope in IAMCredentialsService. For example:

require 'google/apis/storage_v1'
require 'google/apis/iamcredentials_v1'

project_id = 'your-project-id'
service_account_id = 'your-service-account-id'
blob = 'test-blob'

iam_credentials_service = Google::Apis::IamcredentialsV1::IAMCredentialsService.new
iam_credentials_service.authorization = Google::Auth.get_application_default(
  'https://www.googleapis.com/auth/iam'
)

response = iam_credentials_service.sign_service_account_blob(
  "projects/#{project_id}/serviceAccounts/#{service_account_id}",
  Google::Apis::IamcredentialsV1::SignBlobRequest.new(payload: blob)
)

puts "Signed blob: #{response.signed_blob}"

Note that this requires the Service Account Token Creator IAM role to work.