IdentityPoolCredentials does not use POST requests to contact external Identity Provider

[dcardozo]

I'm developing an on-premise service that calls Google's Document AI that uses identity federation for external workloads as recommended by the Best practices for application authentication for On-premises data center . I've successfully tested the workflow using REST APIs; I've been able to get an id token from my on-premise identity provider (Keycloak) and exchange it for an STS token, then an access token and finally invoke the Document AI API.

The problem happens when I try to use the Java client libraries; the call to the identity provider fails with the following error:

Caused by: java.io.IOException: Error getting subject token from metadata server: 405 Method Not Allowed
GET https://HOSTNAME/auth/realms/REALM_NAME/protocol/openid-connect/token

According to the OpenID specifications for Token Requests the expected HTTP method for a token request should be POST. However the section on Obtaining short-lived credentials with identity federation of the Identity and Access Management Guide indicates that an HTTP GET will be issued. The source code for IdentityPoolCredentials.java shows that only the GET method is supported (line 228)

Shouldn't IdentityPoolCredentials support sending POST request to an external Identity Provider to comply with the OpenID specifications?

Here is the content of my credential configuration file generated in the Cloud Console (sensitive data has been removed):

{
  "type": "external_account",
  "audience": "//iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID",
  "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
  "token_url": "https://sts.googleapis.com/v1/token",
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/SERVICE_ACCOUNT_EMAIL:generateAccessToken",
  "credential_source": {
    "url": "https://HOSTNAME/auth/realms/REALM_NAME/protocol/openid-connect/token",
    "headers": {
      "client_id": "CLIENT_ID",
      "client_secret": "CLIENT_SECRET",
      "grant_type": "client_credentials",
      "scope": "openid"
    },
    "format": {
      "type": "json",
      "subject_token_field_name": "id_token"
    }
  }
}

In the above configuration, client_id, client_secret, grant_type and scope are set in the headers, but this is incorrect, as they should be passed as POST data for a successful POST request (I was hoping this would be a hint to use POST instead of GET, but didn't work.)

In the meantime, I tested a successful workaround by switching from url-sourced to file-sourced credentials and save the response from the identity provider in a json file which then IdentityPoolCredentials can use to retrieve the ID token from. It would be much nicer if this extra-step wasn't necessary.

Here is the modified configuration file for a file-sourced credentials:

{
  "type": "external_account",
  "audience": "//iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID",
  "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
  "token_url": "https://sts.googleapis.com/v1/token",
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/SERVICE_ACCOUNT_EMAIL:generateAccessToken",
  "credential_source": {
    "file": "/path/to/response/from/idp/with/id_token.json",
    "format": {
      "type": "json",
      "subject_token_field_name": "id_token"
    }
  }
}

For completeness, here is the error I get in when using url-sourced credentials:

com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:67)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
    at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1133)
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1277)
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
    at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:564)
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:534)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:455)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
    Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
        at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
        at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
        at com.google.cloud.documentai.v1.DocumentProcessorServiceClient.processDocument(DocumentProcessorServiceClient.java:232)
        at com.demo.googledocaipoc.ProcessDocument.processDocument(ProcessDocument.java:57)
        at com.demo.googledocaipoc.GoogleDocaiPocApplication.lambda$docAiRunner$1(GoogleDocaiPocApplication.java:47)
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:760)
        at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:750)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:309)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1303)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1292)
        at com.demo.googledocaipoc.GoogleDocaiPocApplication.main(GoogleDocaiPocApplication.java:29)
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Credentials failed to obtain metadata
    at io.grpc.Status.asRuntimeException(Status.java:535)
    ... 14 more
Caused by: java.io.IOException: Error getting subject token from metadata server: 405 Method Not Allowed
GET https://HOSTNAME/auth/realms/REALM_NAME/protocol/openid-connect/token
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" class="login-pf">

<head>
    <meta charset="utf-8">
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <meta name="robots" content="noindex, nofollow">

            <meta name="viewport" content="width=device-width,initial-scale=1"/>
    <title>Log in to </title>
    <link rel="icon" href="/auth/resources/vknfl/login/keycloak/img/favicon.ico" />
            <link href="/auth/resources/vknfl/login/keycloak/node_modules/patternfly/dist/css/patternfly.min.css" rel="stylesheet" />
            <link href="/auth/resources/vknfl/login/keycloak/node_modules/patternfly/dist/css/patternfly-additions.min.css" rel="stylesheet" />
            <link href="/auth/resources/vknfl/login/keycloak/lib/zocial/zocial.css" rel="stylesheet" />
            <link href="/auth/resources/vknfl/login/keycloak/css/login.css" rel="stylesheet" />
</head>

<body class="">
  <div class="login-pf-page">
    <div id="kc-header" class="login-pf-page-header">
      <div id="kc-header-wrapper" class=""></div>
    </div>
    <div class="card-pf ">
      <header class="login-pf-header">
                <h1 id="kc-page-title">        We are sorry...
</h1>
      </header>
      <div id="kc-content">
        <div id="kc-content-wrapper">

        <div id="kc-error-message">
            <p class="instruction">An internal server error has occurred</p>
        </div>

        </div>
      </div>

    </div>
  </div>
</body>
</html>

    at com.google.auth.oauth2.IdentityPoolCredentials.getSubjectTokenFromMetadataServer(IdentityPoolCredentials.java:242)
    at com.google.auth.oauth2.IdentityPoolCredentials.retrieveSubjectToken(IdentityPoolCredentials.java:188)
    at com.google.auth.oauth2.IdentityPoolCredentials.refreshAccessToken(IdentityPoolCredentials.java:169)
    at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:257)
    at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:254)
    at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
    ... 3 more
Caused by: com.google.api.client.http.HttpResponseException: 405 Method Not Allowed
GET https://HOSTNAME/auth/realms/REALM_NAME/protocol/openid-connect/token
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" class="login-pf">

<head>
    <meta charset="utf-8">
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <meta name="robots" content="noindex, nofollow">

            <meta name="viewport" content="width=device-width,initial-scale=1"/>
    <title>Log in to </title>
    <link rel="icon" href="/auth/resources/vknfl/login/keycloak/img/favicon.ico" />
            <link href="/auth/resources/vknfl/login/keycloak/node_modules/patternfly/dist/css/patternfly.min.css" rel="stylesheet" />
            <link href="/auth/resources/vknfl/login/keycloak/node_modules/patternfly/dist/css/patternfly-additions.min.css" rel="stylesheet" />
            <link href="/auth/resources/vknfl/login/keycloak/lib/zocial/zocial.css" rel="stylesheet" />
            <link href="/auth/resources/vknfl/login/keycloak/css/login.css" rel="stylesheet" />
</head>

<body class="">
  <div class="login-pf-page">
    <div id="kc-header" class="login-pf-page-header">
      <div id="kc-header-wrapper" class=""></div>
    </div>
    <div class="card-pf ">
      <header class="login-pf-header">
                <h1 id="kc-page-title">        We are sorry...
</h1>
      </header>
      <div id="kc-content">
        <div id="kc-content-wrapper">

        <div id="kc-error-message">
            <p class="instruction">An internal server error has occurred</p>
        </div>

        </div>
      </div>

    </div>
  </div>
</body>
</html>

    at com.google.api.client.http.HttpResponseException$Builder.build(HttpResponseException.java:293)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1118)
    at com.google.auth.oauth2.IdentityPoolCredentials.getSubjectTokenFromMetadataServer(IdentityPoolCredentials.java:238)
    ... 9 more

[TimurSadykov]

@lsirac could you please take a look?

[lsirac]

Hey @dcardozo, unfortunately we do not support this. URL based IdentityPoolCredentials are designed to be used with metadata servers (e.g. Azure IMDS). From the documentation: "URL-sourced credentials: Tokens are loaded from a local server with an endpoint that responds to HTTP GET requests. The response must be an OIDC ID token, either in plain text or in JSON format."

I think your best option for now is to continue to save the credential in a file, like you're doing. I know this extra step is tedious. Note you also have to deal with making sure the token stored in the file has not expired.

There's good news though - we're about to add the ability to retrieve the 3P token via an executable (that you define) so you can switch to that once it has been implemented.

[dcardozo]

@lsirac

There's good news though - we're about to add the ability to retrieve the 3P token via an executable (that you define) so you can switch to that once it has been implemented.

Thank you for taking a look at this. I'll keep an eye for that new feature.