IAP rejects requests from other GAE

[AndyClausen]

I've been trying for way too long to get this to work, so I apologize if any rudeness or mistakes come from this:

I'm trying to send a PUT request from one App Engine instance to another App Engine instance. I am getting a client like so:

const client = await auth.getClient();

Correct me if I'm wrong, but that should give me a client with the default GAE service account when run in GAE.

I then send a request like so:

await client.request({
  method: 'PUT',
  url:
    'https://myservice-dot-my-project.appspot.com/api/cool-endpoint?myvar=value',
  data: { some, body, once, told, me },
});

I also tried setting target_audience before the request (although it gives a type error):

client.additionalClaims = {
  target_audience:
    '133742036069-s0m3numb3rs4nd13773rz.apps.googleusercontent.com',
};

With or without the audience claim, I get this thrown in my head: GaxiosError: Invalid IAP credentials: Base64 decode failed on token: ya29.c.<redacted - looks like base64, but weird letters come out when trying to decode>

Probably the most annoying thing is that I can get it to work with auth.fromJSON and providing the json with keys locally. But this is not an option, as the team doesn't want to manage json files for all our apps in all environments.

I have gone through tons of documentation, SO questions, GitHub issues... There's no signs that anyone has seen this issue before - or even tried to use the default service account given by app engine when going through IAP.

Please help, I'm getting desperate.

---

Environment details

• App Engine Flex
• Node.js version: latest
• npm version: latest
• google-auth-library version: latest

Steps to reproduce

1. Make an endpoint of some sort with IAP protection
2. Copy paste code snippets above and insert said endpoint in url
3. Deploy on GAE flex

[AndyClausen]

Also, inb4 "works for me" ™ @JustinBeckwith

[AndyClausen]

@bcoe I'm actually not sure this is a question, it might be a bug. I'm surprised this hasn't been brought up before though, you'd think others have tried sending requests to an IAP protected app from within GCP. Although it might just be me doing something wrong, but I doubt it.

[bcoe]

Hey @AndyClausen, I think you might be able to pull off what you're attempting using firewall rules:

https://cloud.google.com/appengine/docs/standard/python/creating-firewalls

There's a section specifically on the topic of Allowing requests from your services, which specifically speaks to the IP range of App Engine Flex 👍

Let me know if this helps get you on your feet, or whether I've misunderstood your issues.

[bcoe]

(If you're unable to use this approach, I'll need to research the Identity-Aware Proxy a bit, before I can give better advice).

[AndyClausen]

Hey, thanks for the reply

It's definitely not a firewall issue as it works fine with a JSON file with keys. The problem is that it doesn't work with the default credentials. I tried researching IAP as well, and it's a bit of a jungle, haha! But basically, it locks the app behind Oauth, and you need to send to authenticate your request to be able to get through (as I understand it). Now, I would understand if you wouldn't be able to send requests with this auth library to an IAP protected app - then it'd just be a missing feature or something. The reason why I think it's a bug, is because it works using a JSON file to make the auth client - but not when making a client from the default application client.

Again though, I might be misunderstanding how it all works.

[bcoe]

@AndyClausen apologies for not having gone too deep on this feature either :smile:

When using the default client, it should be using what ever grants you've given to YOUR_PROJECT_ID@appspot.gserviceaccount.com, have you given this service account in IAM, IAP permissions (phew, that's a lot of acronyms).

[AndyClausen]

Ikr, ever since we started using GCP every meeting have been half acronyms half actual words haha!

But yeah, it has the correct permissions. The JSON I used which worked was for the same service account.

[AndyClausen]

I just noticed you marked as feature request - does that mean that this is intended? That you are not meant to be able to communicate with applications protected with IAP without a JSON key file?

[bcoe]

@ace-n any thoughts on this one, have you used IAP and Google App Engine in conjunction?

[AndyClausen]

You still have needs more info here. Is there any more info I can give?

[bcoe]

@AndyClausen I think we have enough info :+1: I have some good news, which is that we have some folks on the team starting to specifically work on auth related issues ... I will point them in the direction of this.

[AndyClausen]

Awesome! We have a couple of apps waiting for this, currently using JSON files to Auth with IAP. Waiting in excitement for it, the team will be happy to hear that <3

[victorbadila]

Hey there, any update on this? :)

[bcoe]

@victorbadila I'm working with @bshaffer this afternoon on a feature that should start to add better support to our auth library for IAP, no guaranteed timeline for when this work lands, but it's in progress.

[bshaffer]

With the code in master, you can now do this:

const client = await auth.getIdTokenClient(
  '133742036069-s0m3numb3rs4nd13773rz.apps.googleusercontent.com'
);
await client.request({
  method: 'PUT',
  url: 'https://myservice-dot-my-project.appspot.com/api/cool-endpoint?myvar=value',
  data: { some, body, once, told, me },
});

And as long as you've set the GOOGLE_APPLICATION_CREDENTIALS environment variable (or are making the call from GCE/AppEngine/GKE/Cloud Run), the client will populate the Authorization header with an ID token.

[bcoe]

:wave: @AndyClausen this is now released to npm; closing this issue, but please feel free to reopen if you bump into any problems with the implementation -- excited to have you try it out.

[AndyClausen]

Thanks a lot! I'll try to make time for it soon.

[AndyClausen]

A team member just tried using this new functionality - it gives the following error: Invalid IAP credentials: JWT 'email' claim isn't a string

Is this because we need to specify the SA email somewhere? Shouldn't it get that by itself?

EDIT: Just to clarify, this was tested through app engine.

[bcoe]

@AndyClausen, @bshaffer can hopefully provide additional clarity, but I think you would need to specify an email, along these lines:

const client = auth.getIdTokenClient('xyz@appspot.gserviceaccount.com');

where xyz@appspot.gserviceaccount.com is your App Engine service account in IAM, if this doesn't work perhaps there's a bug we need to address.

[AndyClausen]

@bcoe That's supposed to be the target audience, in this case the IAP client. The email is supposed to be in the environment already AFAIK.

[bcoe]

@AndyClausen if you provide the email explicitly does it work, wondering if this is an improvement we should make to the current implementation, or whether you're currently completely blocked.

[salrashid123]

i think iap looks for the email field in in the token. if you use a json service accont, the token is populated

if you run the following snippet in a gce instance, it'll fail against IAP

async function mainIAP() {
  const u = 'https://bmineral-minutia-820.appspot.com';
  const a = '1071284184436-en7pnh3e250v6p2r0ekredacted.apps.googleusercontent.com';
  const auth = new GoogleAuth();
  const client = await auth.getIdTokenClient(
    a
  );
  const res = await client.request({
    method: 'GET',
    url: u,
  });
  console.log(res.data);
}

will error with

 data: "Invalid IAP credentials: JWT 'email' claim isn't a string",

however, if you edit the following bit inline and add on &format=full (which you can read about here),

https://github.com/googleapis/google-auth-library-nodejs/blob/master/src/auth/computeclient.ts#L112

then an id token from gce will work against iap (a service account json file will work since the exchanged id_token has it already).

---

now, i don't know if the token returned by cloud run or gae v2 (or gcf) has the email part in it (you can verify by printing the id token and decoding it at jwt.io)

[bshaffer]

Ahh I found a similar issue here, https://github.com/cloudendpoints/esp/issues/675, and indeed the way it was fixed is by including format=full in the metadata request. This seems to only effect ID tokens retrieved from GCE.

I'll add format=full to the metadata querystring, and this should fix the issue.

[salrashid123]

While format=full will incude the email claim, it'll also throw in a whole bunch of other stuff too like configuration of the GCE instance (which is gonna expose more data in the easily decodeable jwt token than is necessary. I'm going to file an internal bug to see if the IAP check can use the sub filed in the claim:

if i go on a GCE instance and run

$ curl -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=1071284184436-en7pnh3e250v6p2r0ekue05ekg30vfbh.apps.googleusercontent.com"

i'll get a jwt that i can decode at jwt.io:

{
  "aud": "1071284184436-en7pnh3e250v6p2r0ekue05ekg30vfbh.apps.googleusercontent.com",
  "azp": "100147106996764479085",
  "exp": 1579189692,
  "iat": 1579186092,
  "iss": "https://accounts.google.com",
  "sub": "100147106996764479085"
}

the sub value is actually the encoded email for the GCE instance's service account. THat value is easily decoded by any number of other systems at the perimeter like Cloud Run, Cloud Functions, etc....its jus that IAP seems to look for plain email as the claim.

I'll cc you on the internal bug but my 2c is holding off on the change above to make &format=full default for now (because if IAP can look for sub, then the code youv'e got here would work)...also, i can't confirm what claims the token has in cloud run, gcf, etc (i'll ask a collegue now about that)

[AndyClausen]

Just FYI: IAP uses both email and sub.

[bcoe]

While format=full will incude the email claim, it'll also throw in a whole bunch of other stuff too like configuration of the GCE instance

@bshaffer what do you propose we do here, should we revert this change; this data will be coming via an HTTPS connection, so the additional JWT information will only be within the process performing the authentication, but I could understand wanting to be opt in to this.

[AndyClausen]

... is there other options than full? Perhaps there's something in the code behind IAP that can help with this?

[bshaffer]

I think it would be best, unfortunately, to leave the integration between GCE and IAP broken for now until we find a better fix for the issue, as packing additional data in the ID token is definitely not a great solution.

[salrashid123]

One workaround if you absolutely need to use a GCE instance is to invoke the IAM API generateIdToken

its an awkward flow where a service account uses an API "on itself" to sign something or to get an id_token.

I woudn't recommend this as a long term solution (its far better if gce can return the email and/or IAP accepts and decodes the sub claim)

Anyway, to use this flow, you'll need to manaully do the exchange since this isn't included in the library to directly do the exchange (though the IAM api is the basis for this pr for impersonated credentials )

export TOKEN=`curl -s -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" | jq -r '.access_token'`export SA_EMAIL=`curl -s -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email"`
export SA_EMAIL=`curl -s -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email"`

curl -s -H "Authorization: Bearer $TOKEN" \ 
   --header 'Content-Type: application/json' \
   -d '{  "audience": "https://foo.bar", "includeEmail": "true" }'\  
  https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/$SA_EMAIL:generateIdToken

[sofisl]

@salrashid123 After connecting with @bcoe and @bshaffer on this issue, it looks like there's an internal bug filed, we're waiting for a release on it.

@AndyClausen, will this solution work for you in the meantime (adding the format=full option)? As mentioned above, we are expecting a release that would pare down these fields eventually.

[AndyClausen]

I guess so. It shouldn't be an issue for us since it's all internal communication :)

[SurferJeffAtGoogle]

@bshaffer Can we close this issue?

[bshaffer]

Yes!

[AndyClausen]

Gotchu