[Storage] Easily generate signed URLs when using workload identity

[jacobo]

Hi, I am open to writing a PR to make code changes for this feature but wanted to start with an issue to get feedback.

When developing locally I can generate signed URL simply with:

require "google/cloud/storage"
storage = Google::Cloud::Storage.new(
  project_id: "<project_id>",
  credentials: ENV["GCP_CREDENTIALS_PATH"]
)
bucket = storage.bucket "<bucket_name>"
remote_file = bucket.file "foo/bar", skip_lookup: true
url = remote_file.signed_url method: "PUT", content_type: "application/json"

However, when running my code from within an eks pod with access grant via workload identitiy (https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity)

I get the error:

/usr/local/bundle/gems/google-cloud-storage-1.31.1/lib/google/cloud/storage/file/signer_v2.rb:88:in `determine_issuer': Service account credentials 'issuer (client_email)' is missing. To generate service account credentials see https://cloud.google.com/iam/docs/service-accounts (Google::Cloud::Storage::SignedUrlUnavailable)

Based on https://github.com/salrashid123/gcpsamples/blob/master/gcs_keyless_signedurl/main.py I was able to write some equivalent ruby code that works for generating a working signed URL (not the most well-factored code, just a proof-of-concept)

require "google/cloud/storage"
storage = Google::Cloud::Storage.new
bucket = storage.bucket "<bucket_name>"
remote_file = bucket.file "foo/bar", skip_lookup: true
path = "/#{bucket.name}/#{remote_file.name}"
url_manually = "https://storage.googleapis.com#{path}"
access_token = JSON.parse(`curl -s -H 'Metadata-Flavor: Google' http://metadata/computeMetadata/v1/instance/service-accounts/default/token`)["access_token"]
connection = Faraday.new(url: "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/#{service_account_email}:signBlob") do |conn|
  conn.request :json
end
expiration = 1.day.from_now.to_i
verb = "PUT"
content_md5 = ""
content_type = "application/json"
signature_string = "#{verb}\n#{content_md5}\n#{content_type}\n#{expiration}\n#{path}"
response = connection.post do |req|
  req.headers["Authorization"] = "Bearer #{access_token}"
  req.body = {delegates: [], payload: Base64.encode64(signature_string).gsub("\n","")}
end
signed_blob = JSON.parse(response.body)["signedBlob"]
signed_url = url_manually + "?" + URI.encode_www_form("GoogleAccessId": service_account_email, "Expires": expiration, "Signature": signed_blob)

So... I was thinking perhaps we could/should just change the implementation of signed_url to detect the conditions under which key-based signing will fail and automatically attempt this meta data service token + signBlob service method.

Or perhaps it's already exposed somewhere else within the depths of this ruby gem? (in which case in the very least the error message you see could be updated to point that out)

Thanks, Jacob

---

Update based on https://googleapis.dev/ruby/google-cloud-storage/latest/Google/Cloud/Storage/File.html#signed_url-instance_method I have a better version

require "google/cloud/storage"
storage = Google::Cloud::Storage.new
bucket = storage.bucket "<bucket_name>"
remote_file = bucket.file "foo/bar", skip_lookup: true
issuer = "<service account email>"
signer = lambda do |string_to_sign|
  IAMCredentials = Google::Apis::IamcredentialsV1
  iam_client = IAMCredentials::IAMCredentialsService.new
  scopes = ["https://www.googleapis.com/auth/iam"]
  iam_client.authorization = Google::Auth.get_application_default scopes
  request = Google::Apis::IamcredentialsV1::SignBlobRequest.new(
    payload: string_to_sign
  )
  resource = "projects/-/serviceAccounts/#{issuer}"
  response = iam_client.sign_service_account_blob resource, request
  response.signed_blob
end
signed_url = remote_file.signed_url method: "PUT", content_type: "application/json", issuer: issuer, signer: signer

And the only things that are "custom" to my use case in this example end up being arguments into the signed_url method. so perhaps a simplification this gem could make would be to auto-generate the signer proc

Finally, I need a way to tell which version of the signed_url method args to use.

When running locally I have a credentials JSON file. so I need only call remote_file.signed_url method: "PUT", content_type: "application/json" Whereas from my pod I have to call remote_file.signed_url method: "PUT", content_type: "application/json", issuer: issuer, signer: signer

If I do it backwards (specify an issue and signer when not on a pod) I get

/Users/jacob/.gem/ruby/3.0.1/gems/googleauth-0.16.2/lib/googleauth/application_default.rb:76:in `get_application_default': Could not load the default credentials. Browse to (RuntimeError)
https://developers.google.com/accounts/docs/application-default-credentials
for more information

Is there some simple call I could make to this gem to determine what context I'm in?

[jacobo]

looks like there is a related open feature request at https://github.com/googleapis/google-auth-library-ruby/issues/270

[quartzmo]

Is there some simple call I could make to this gem to determine what context I'm in?

@jacobo Have you tried using ruby-cloud-env? (It's already a dependency of google-cloud-storage via google-cloud-core.)

For example:

require "google/cloud/env"
env = Google::Cloud.env
env.kubernetes_engine?

[quartzmo]

Background: The signer parameter and examples showing sign_service_account_blob usage were added in PR #7091.

[frankyn]

Current state of FR(1/16/23): In #7091, a new signer parameter was introduced to allow passing in a method to create signature from a payload to support signed URLs. However, it still requires boiler plate code to check the environment a client is in (GCE or local dev env) to determine how to call #signed_url when generating a Signed URL.

To address this issue:

1. Add an option to auto-detect issuer and key using Google::Cloud.env#lookup_metadata
2. Add support for calling IAMCredentials#sign_service_account_blob when in GCE environment Google::Cloud.env#compute_engine?
3. Update guidance on GCE environments in documentation

These steps should reduce overhead of boiler plate code using signer parameter in signed_url and at the same existing signer / issuer support provides an escape hatch for more complex situations that can't be auto-detected.

Workaround for now:

require "google/cloud/env" # Dependency of google-cloud-storage gem
require "google/cloud/storage"
require "google/apis/iamcredentials_v1"
require "googleauth"

storage = Google::Cloud::Storage.new

bucket = storage.bucket "my-todo-app"
file = bucket.file "avatars/heidi/400x400.png", skip_lookup: true

env = Google::Cloud.env
if env.compute_engine?
  # Issuer is the service account email that the Signed URL will be signed with
  # and any permission granted in the Signed URL must be granted to the
  # Google Service Account.  
  issuer = env.lookup_metadata "instance", "service-accounts/default/email"

  # Create a lambda that accepts the string_to_sign
  signer = lambda do |string_to_sign|
    IAMCredentials = Google::Apis::IamcredentialsV1
    iam_client = IAMCredentials::IAMCredentialsService.new

    # Get the environment configured authorization
    scopes = ["https://www.googleapis.com/auth/iam"]
    iam_client.authorization = Google::Auth.get_application_default scopes

    request = Google::Apis::IamcredentialsV1::SignBlobRequest.new(
      payload: string_to_sign
    )
    resource = "projects/-/serviceAccounts/#{issuer}"
    response = iam_client.sign_service_account_blob resource, request
    response.signed_blob
  end

  url = file.signed_url method: "GET", issuer: issuer,
                      signer: signer
else
  url = file.signed_url method: "GET"
end

References:

1. https://github.com/googleapis/google-auth-library-ruby/issues/270#issuecomment-934987654