Support AppEngine Flex in custom image mode based on ESPv2 image

[maroux]

Hi

I'm trying to deploy esp in the following configuration:

ESPv2 - gcr.io/endpoints-release/endpoints-runtime-serverless:2.10.0 deployed to App Engine flex with env:

GOOGLE_APPLICATION_CREDENTIALS = "<service account JSON path>"
ENDPOINTS_SERVICE_NAME = "<service>-dot-<project>.appspot.com"

Endpoints config: Copy of sample with these changes:

-host: "YOUR-PROJECT-ID.appspot.com"
+host: "<service>-dot-<project>.appspot.com"

-    x-google-audiences: "YOUR-CLIENT-ID"
+    x-google-audiences: "<actual IAP client id>"

+  auth0_jwk:
+    authorizationUrl: ""
+    flow: "implicit"
+    type: "oauth2"
+    x-google-issuer: "https://<tenant>.auth0.com/"
+    x-google-jwks_uri: "https://<tenant>.auth0.com/.well-known/jwks.json"
+    x-google-audiences: "https://<esp service>-dot-<project>.appspot.com/"
+
+x-google-backend:
+  address: "https://<service>-dot-<project>.appspot.com"
+  jwt_audience: "<IAP oauth client id>"
+  protocol: h2
+
+
+x-google-endpoints:
+  - name: <service>-dot-<project>.appspot.com

The app is fronted by IAP to restrict access only to the ESP user. However, the requests fail with this error:

HTTP/2 401 
date: Sat, 30 May 2020 22:38:06 GMT
content-type: text/html; charset=UTF-8
content-length: 57
x-goog-iap-generated-response: true
x-envoy-upstream-service-time: 27
server: envoy
via: 1.1 google
alt-svc: h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

Invalid IAP credentials: JWT 'email' claim isn't a string

Upon debugging, I found the token that ESP sends its requests with looks like this:

{
  "aud": "<IAP oauth client id>",
  "azp": "103551354333791234484",
  "exp": 1590879029,
  "iat": 1590875429,
  "iss": "https://accounts.google.com",
  "sub": "103551354333791234484"
}

As seen here, sub isn't the email of the service account I specified in ESP configuration.

Please advise.

[qiwzhang]

Since you are using IAP, I suspected that ESPv2 is receiving IAP token in x-goog-iap-jwt-assertion header, ESPv2 used it instead of the one in "authorization" header.

You can work around this problem by using x-google-jwt-locations field, to only specify "Authorization" header, not to include "x-goog-iap-jwt-assertion" header

[maroux]

That error is from the IAP connection downstream from ESP actually. The IAP in front of ESP works just fine and ESP doesn't prefer IAP header.

The problem is in the JWT created by ESPv2 to talk to the backend.

[nareddyt]

Here is an example JWT token that ESPv2 generates using IMDS. This was generated for a Cloud Run backend, but the logic should be the same.

eyJhbGciOiJSUzI1NiIsImtpZCI6Ijc5YzgwOWRkMTE4NmNjMjI4YzRiYWY5MzU4NTk5NTMwY2U5MmI0YzgiLCJ0eXAiOiJKV1QifQ.eyJhdWQiOiJodHRwczovL3B5dGhvbi1ncnBjLWJvb2tzdG9yZS1zZXJ2ZXItbHJubTNlanBlcS11Yy5hLnJ1bi5hcHAiLCJhenAiOiIxMDUyODY5MDc4NTU2OTE4MzIyODUiLCJlbWFpbCI6IjcxMDcyNjU4OTE1MC1jb21wdXRlQGRldmVsb3Blci5nc2VydmljZWFjY291bnQuY29tIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsImV4cCI6MTU4MjY3ODUwNywiaWF0IjoxNTgyNjc0OTA3LCJpc3MiOiJodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20iLCJzdWIiOiIxMDUyODY5MDc4NTU2OTE4MzIyODUifQ.Bpls9L3gqG8Xc_3ag8ogTAVCHjwgcbMHVVsvwnN_9YalMfarrcvqbEw9nvZ0m7rICJloGyxA3hX_n_p883vCoLbvYiqD9qk5tAPWja0P8pVY0oVBjMt-WDcfHqehLxueCQjhbPykPZkNXy0LSzqOyp_SlPt2ku9W83woCkwymbvVCFNwy_Hy-HbmsWa5qrCuxc1RWhoKrhMrFYHZ9Ub2cPqUL6Xx9QpSM7ngKLP1ygPVHgFW5hajCfzdMt1NTvMCzt_KvSDmhi8HFAPslFCeMp63-ZuaTPzG3n3lDaa3rTgCUaqhdO124wio902KjsIbGxLf8ZbYGH8D3GrYP0aLOQ

The payload decodes to:

{
  "aud": "https://python-grpc-bookstore-server-lrnm3ejpeq-uc.a.run.app",
  "azp": "105286907855691832285",
  "email": "710726589150-compute@developer.gserviceaccount.com",
  "email_verified": true,
  "exp": 1582678507,
  "iat": 1582674907,
  "iss": "https://accounts.google.com",
  "sub": "105286907855691832285"
}

It seems like the token you posted above is missing the email claim. This is not related to the sub claim.

Are you 100% sure that is the token that ESP is sending to your app engine backend? If so, @qiwzhang any ideas why the token generated in the original issue is missing the email field?

[qiwzhang]

we are using Metadata service to get ID token,

Could you try to use

curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/identity

To see what token you are getting.

You need need to ssh into ESP container to run that, not sure if it is possible.

[maroux]

@nareddyt I got that from ESPv2 logs. Confirmed it with a hello world app that dumps headers as well.

@qiwzhang I had to add audience query param, but yes, this is what the result looks like (run from ssh into App Engine box):

{
  "aud": "<IAP oauth client id>",
  "azp": "103551354333791234484",
  "exp": 1591039037,
  "iat": 1591035437,
  "iss": "https://accounts.google.com",
  "sub": "103551354333791234484"
}

@qiwzhang even with GOOGLE_APPLICATION_CREDENTIALS set - ESPv2 uses compute metadata service?

[maroux]

Per IAP docs looks like there's a different way to get the token needed for authenticating against IAP?

[maroux]

Update: Adding ?format=full to the metadata endpoint works and returns this:

{
  "aud": "<IAP oauth client id>",
  "azp": "103551354333791234484",
  "email": "<project>@appspot.gserviceaccount.com",
  "email_verified": true,
  "exp": 1591039587,
  "google": {
    "compute_engine": {...}
  },
  "iat": 1591035987,
  "iss": "https://accounts.google.com",
  "sub": "103551354333791234484"
}

Of course, this doesn't generate the token using the service credentials I provided using GOOGLE_APPLICATION_CREDENTIALS.

[qiwzhang]

It seems that this is a new feature request, to generate ID token for a backend protected by IAP with client_id. Currently, we only generate ID token for backend service protected by IAM with audience.

Your link has many go/python/java sample codes of using google.oauth2 package. We need to study to find out specific HTTP API of GCP metadata server and using c++ code to implement it.

[qiwzhang]

Hi Jason, could you take a look on this feature request?

[qiwzhang]

It seems that there are two issues here: 1) "email" field in ID token payload is needed for IAP to work. From AppEngine Flex Metadata server, we have to append "?format=full" to get it. In Cloud Run, it is not needed. Should we always append it in order to support AppEngine Flex, in this special deployment.

2) If environment variable GOOGLE_APPLICATION_CREDENTIALS is set. should we use its service account file to get ID token? Currently, we always use the service account deployed ESPv2. I am not sure if we should support this feature.

[qiwzhang]

This could be a way to support ESPv2 in AppEngine Flex with custom image as long as backend code can be run in ESPv2 image. 1) build a custom docker image based on ESPv2 image, copy all backend application codes into this image, add entrypoint script to run ESP start_proxy.py and application code. 2) deploy such app with that docker image.

ESPv2 will be running in the same container as backend application, but in two different processes. AppEgine Flex has a side-car container running Nginx. the request will go through this Nginx first, the ESPv2, then backend.

But this is still better than running ESPv2 in a remote proxy in Cloud Run.

The drawback is: application code has to be able to install run in ESPv2 image. They have to use custom run-time, could not use other standard run-times, such as Java, Python.

This is just a proposal. It needs to be tested. E.g. the ID token problem need to be fixed.

[maroux]

ESPv2 will be running in the same container as backend application, but in two different processes.

Yep that's what I ended up doing. Except instead of putting application code inside ESP docker, I did the opposite - put ESP code inside application Docker. In addition, endpoints config disables auth because its simply talking to localhost.

If you want to document it, here's what it looks like:

FROM gcr.io/endpoints-release/endpoints-runtime-serverless:{{ ESP_VERSION }} AS esp
...
# Copy ESP resources from ESP docker stage
COPY --from=esp /apiproxy /apiproxy
COPY --from=esp /bin /bin
COPY --from=esp /env_start_proxy.py /bin/env_start_proxy.py
RUN chmod +x /bin/*

ENV PATH=/bin:$PATH \
    ESPv2_ARGS="..."

[qiwzhang]

Thanks. @maroux That is a cool idea to just copying ESP binary into your application docker image. ESP binary is just one c++ binary and some python codes, they should be able to run on many Linux runtime.

[qiwzhang]

I have done some prototyping. Here are the key issues I found with this approach:

1) package compatibility issue: we are trying to run two components in one VM; ESPv2 and your app. they may not compatible with each others. If the docker image is based on ESPv2 docker image, your application may not work and may need to install a lot of packages. If the docker image is based on the application docker image, ESPv2 may not work. ESPv2 envoy binary is a c+++ compiled one, it may not work for some latest Linux OS. For example, it doesnt work with gcr.io/google-appengine/python, complaint about glibc missing.

2) Docker entrypoint: your application needs a script to start its process, and ESPv2 needs one to start its processes too. But Docker entrypoint only allows one script. May need to write a new script to start both.

Found python path is hardcoded here. It may not work for some OS.