SPIRE deployments of more than a few servers can create more keys in JWKS than OIDC federating system supports

[keeganwitt]

Using the KMS KeyManager plugin, I currently see 8 keys in my testing (although this might have been caused by pods being replaced without the key_metadata_file being in a persisted volume). Also in my testing, AWS support a maximum of 100 keys in a JWKS endpoint, otherwise it is unable to retrieve the KID from the list to match against the one used in a JWT. Since I have 15 clusters with 2 server instances each, if 4 keys are created each, I'll get 120 keys created, which means I won't be able to use SPIRE for OIDC federation with AWS.

[amartinezfayo]

Using the KMS KeyManager plugin, I currently see 8 keys in my testing (although this might have been caused by pods being replaced without the key_metadata_file being in a persisted volume)

We have an open issue to remove the dependency on the key_metadata_file that KMS plugins rely on: https://github.com/spiffe/spire/issues/4375.

[keeganwitt]

Using the KMS KeyManager plugin, I currently see 8 keys in my testing (although this might have been caused by pods being replaced without the key_metadata_file being in a persisted volume)

We have an open issue to remove the dependency on the key_metadata_file that KMS plugins rely on: #4375.

I'll leave this ticket open for now so I can validate, but that will likely resolve my concern (assuming 2 keys per server instance).

[keeganwitt]

Confluent Cloud only supports 15 keys per identity provider. I'm discussing with them whether they can make an exception and what that new limit might be.

[amartinezfayo]

Also in my testing, AWS support a maximum of 100 keys in a JWKS endpoint, otherwise it is unable to retrieve the KID from the list to match against the one used in a JWT.

Is there any chance that there is a resource quota establishing the limit to 100 keys? It seems that the default limit is 100,000 keys: https://docs.aws.amazon.com/kms/latest/developerguide/resource-limits.html#kms-keys-limit.

I'll leave this ticket open for now so I can validate, but that will likely resolve my concern (assuming 2 keys per server instance).

Just to clarify: SPIRE publishes 4 keys (current X509 CA, next X509 CA, current JWT authority, next JWT authority), so there will be 4 keys per server instance. I remember that you mentioned during the last contributors sync that you are only interested in the keys exposed by the SPIRE OIDC Discovery Provider? I may be missing something. What I'm not quite sure is how that could limit the number of keys pushed to AWS KMS by SPIRE Server.

[keeganwitt]

The link you mentioned is about KMS keys. The key limit I'm talking about is when you create an OpenID Identity Provider in AWS and point it to SPIRE's OIDC component, the /keys endpoint can't contain more than 100 keys or else AWS will be unable to find the key to use in JWT validation. This is a limit that is not publicly disclosed and only found by my manual testing.

Yes by "2 keys" I meant what's exposed by the OIDC provider, not the total keys each instance adds to the bundle. I think this is something we confirmed already in the contributor meeting.

[amartinezfayo]

The key limit I'm talking about is when you create an OpenID Identity Provider in AWS and point it to SPIRE's OIDC component

Thanks for the clarification, it does make sense to me now.

So we are keeping this open until #4700 is done and you confirm that this is no longer a concern for you?

[keeganwitt]

So we are keeping this open until https://github.com/spiffe/spire/pull/4700 is done and you confirm that this is no longer a concern for you?

I think given what Confluent told me (still waiting to find out how much they can increase the limit), we should keep this open regardless and discuss what options we have when one has to federate with a system with a lower JWKS key limit than the SPIRE deployment requires. There may be other systems I or others have to integrate where this limit is hit and the vendor refuses to increase it.

[azdagron]

Unfortunately, there isn't an immediately clear way to reduce the number of JWT signing keys in use in SPIRE HA environments (two per server is a hard-coded behavior).

It would be good to explore solutions for using SPIRE with OIDC validators with these constraints.

[keeganwitt]

One idea is to add a SPIRE cluster/deployment identifier in the keys in the bundle so that each cluster can filter the bundle the only keys that are from that particular cluster/deployment and configure each cluster/deployment with a separate issuer to know which OIDC endpoint to hit. Would this be a viable option conceptually?

[mcdafydd]

I've been curious about the idea of including the signing and any intermediate public keys as certs using the x5c field in JWT-SVID headers and allowing the JWKS endpoint to just serve a root certificate. It's a bit similar to the idea of distributing intermediate certs during TLS hello messages, albeit with a different trust model since the root certs for TLS are typically pre-installed versus being served from an HTTPS server managed by the same organization generating JWT-SVIDs.

The JWT-SVID standards don't allow for this today, and I'm not yet clear how many providers we might federate with support x5c when validating JWTs. x5c seems mentioned along with jwk and jku in regards to key confusion attacks, but I haven't located any details on cases where this was exploited publicly.

[keeganwitt]

Notes from our contributors sync meeting:

Not all OIDC providers support the x5c field. Evan thought AWS didn't support that. Additionally, the SPIRE team felt that including the signing key in the JWT itself wasn't safe.

It was my interpretation of the spec that it was required the signing key be in the JWKS endpoint and the x5c validation was optional. Evan mentioned he has seen bespoke implementations that implemented things in this way, but that's only possible if you control the federating system as well (which we don't in this case since it's an external vendor).

[evan2645]

One idea is to add a SPIRE cluster/deployment identifier in the keys in the bundle so that each cluster can filter the bundle the only keys that are from that particular cluster/deployment and configure each cluster/deployment with a separate issuer to know which OIDC endpoint to hit. Would this be a viable option conceptually?

I think this would be functionally equivalent to running multiple SPIRE clusters that aren't nested?

[keeganwitt]

One idea is to add a SPIRE cluster/deployment identifier in the keys in the bundle so that each cluster can filter the bundle the only keys that are from that particular cluster/deployment and configure each cluster/deployment with a separate issuer to know which OIDC endpoint to hit. Would this be a viable option conceptually?

I think this would be functionally equivalent to running multiple SPIRE clusters that aren't nested?

As far as JWTs are concerned, yes, I think effectively this would remove the nesting. But this idea wouldn't affect X.509s which could continue to operate nested. This idea seems pretty hacky...

[kfox1111]

Another option might be for the root server to generate the keys, and then push them down to all child servers, to minimize the number of keys. Not ideal for all use cases, so would probably want to support the existing model too and make it conditional?

[keeganwitt]

Another option might be for the root server to generate the keys, and then push them down to all child servers, to minimize the number of keys. Not ideal for all use cases, so would probably want to support the existing model too and make it conditional?

I had the same thought. This is the best way to support this that I can think of.

[keeganwitt]

Another option (although not as secure) would be a static key, which could be configured in the CA manager. Possibly by making the CA manager a new plugin type or adding a JwtKeyProvider plugin type.