TriggerAuthentication Support for External Scalers

[wsugarman]

Proposal

TriggerAuthentication objects are a convenient way to share, and use identities for, accessing resources required by the scaler. However, external scalers can't easily take advantage of them. My proposal would be to add support for TriggerAuthentication in external scalers, perhaps by updating the gRPC service contract to include an authentication token from the TriggerAuthentication configured in the ScaledObject.

Use-Case

A single external scaler service may access different cloud resources based on different ScaledObject metadata. For example, today an azure-servicebus trigger may be used to access different queues in different subscriptions altogether. While the built-in scalers can use multiple TriggerAuthentication objects to access different resources, external scalers must either:

1. Include credentials in the ScaledObject
2. Ensure a managed identity with access to both resources is present in the gRPC service pod

Instead, it would be much easier to be able to re-use existing TriggerAuthentication objects like other scalers.

[zroubalik]

@wsugarman no longer an issue?

[wsugarman]

@zroubalik - Well, I think I misunderstood how TriggerAuthentication works with managed identities like AAD Pod Identity. If my reading is correct, if I had a TriggerAuthentication resource like the following:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: my-auth
spec:
  podIdentity:
    provider: azure

Its usage in a ScaledObject simply tells the built-in scaler implementations to assume there is an AAD identity found in the pod to use with sufficient access. Similarly, we could create an AAD Pod Identity Binding for a label assocaited with the external scaler instead of KEDA for the same effect. Furthermore, I didn't consider the other authentication types like secrets, environments, and hashicorp vault.

It would be nice however to support multiple managed identities for KEDA, such that a TriggerAuthentication can reference an identity via its ClientId. On the other hand, including 5 managed identities in a pod with different roles seems to be semantically equivalent to including 1 managed identity with all of the roles on the same pod. Perhaps this could be its own feature.

Nevertheless, I could re-activate this feature as it would be nice for a consistent representation of permissions for both the built-in and external scalers, but I think the above proposal needs to be fleshed out more.

[porscheme]

Why was this closed?

[wsugarman]

@porscheme - Ultimately I was unsure what the actual proposal would look like. How exactly would an external scaler use the TriggerAuthentication resource? Would it just be metadata that is a standard part of the gRPC scaler API instead of being included by the author as part of the ScaledObject resource?

[porscheme]

@porscheme - Ultimately I was unsure what the actual proposal would look like. How exactly would an external scaler use the TriggerAuthentication resource? Would it just be metadata that is a standard part of the gRPC scaler API instead of being included by the author as part of the ScaledObject resource?

It should be part of the standard gRpc metadata. Because, KEDA gRpc client would use this info to get a AccessToken (DefaultAzureCredential) and send it as meadata when calling external scaler!

[wsugarman]

Oh interesting. To be clear, are you proposing that the KEDA pods would fetch a token, like the built-in scalers, and then send the token as part of the external scaler gRPC request?

[porscheme]

Oh interesting. To be clear, are you proposing that the KEDA pods would fetch a token, like the built-in scalers, and then send the token as part of the external scaler gRPC request?

Exactly, would that be any problem? This design follows KEDA's secure by default architecture design principle. Current external scaler solution is bit pathetic, not sure if anyone using it a meaning full way.

We currently use PodIdentity in ScaleObject, ScaleJob, azureKeyVault in objects.

[wsugarman]

@porscheme I think it makes sense from a usability perspective, and it would definitely unify the authentication story.

I'm only apprehensive about sending the token, a sensitive piece of data, over the K8s network. If a user hasn't configured TLS (which is the default), then it's not encrypted! In KEDA, TLS relies on the external scaler to have implemented it themselves, as the KEDA pod can only optionally validate the certificate authority today.

@zroubalik What do you think?

[porscheme]

@porscheme I think it makes sense from a usability perspective, and it would definitely unify the authentication story.

I'm only apprehensive about sending the token, a sensitive piece of data, over the K8s network. If a user hasn't configured TLS (which is the default), then it's not encrypted! In KEDA, TLS relies on the external scaler to have implemented it themselves, as the KEDA pod can only optionally validate the certificate authority today.

@zroubalik What do you think?

Your reply bit confusing... SSL: Only Server authentication (traffic on the wire is encrypted) TLS : Mutual Certificate Authentication this requires SSL

• External scaler SHOULD be over SSL, may not be TLS (Certificate Authentication) enabled
• If user hasn't implemented SSL on the scaler, its error condition

[wsugarman]

Oh, my mistake! I see in the past couple months the external scaler code path was updated to support TLS: https://github.com/kedacore/keda/issues/3565. That is great news! Still, the default for KEDA is an plain old HTTP connection between the KEDA pod and the external scaler pods.

That being said, perhaps you're right and KEDA could reject the usage of TriggerAuthentication for external scalers when the connection is unencrypted to preempt potential security issues.

Nevertheless, I'll re-open the issue as there is some good discussion going on.

[porscheme]

Thanks, would love to be first user :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[tomkerkhove]

I was looking for this one to give it a nudge, so no stale bot - We cannot close :)

[tomkerkhove]

@kedacore/keda-maintainers We need a solid story for external scalers to be able to plug in to the existing TriggerAUthentication sources and pull in respective secrets.

While doing that, we need to put in security measures to prevent "rogue" external scalers from extracting and leaking secrets.