Proposal: Extend service invocation to non-Dapr endpoints

[sicoyle]

Formalizing the discussion for the external service invocation feature here. Additional context may be found in the initial issue.

[yaron2]

Thanks for the prompt responses to requested changes @sicoyle

+1 binding

[daixiang0]

Overall is good, +1

[artursouza]

Please, pick an option. My recommendation is to make it HTTP-specific. We had tried creating a generic layer for HTTP and gRPC but it ends up with odd attributes that are only applicable to one.

[sicoyle]

Please, pick an option. My recommendation is to make it HTTP-specific. We had tried creating a generic layer for HTTP and gRPC but it ends up with odd attributes that are only applicable to one.

I updated the doc selecting the HTTP-specific approach, while dropping the redundant protocols field. Please let me know if anyone has other thoughts on this!

[ItalyPaleAle]

Sorry I didn't see this sooner.

One comment/question I have is whether it'd be possible to offer a programmatic way to define these external endpoints too, in addition to using a CRD (like we do for pubsub subscriptions for example).

Also, how would this interact (if it would at all) with mTLS? And how could callers be authenticated?

[yaron2]

Sorry I didn't see this sooner.

One comment/question I have is whether it'd be possible to offer a programmatic way to define these external endpoints too, in addition to using a CRD (like we do for pubsub subscriptions for example).

Also, how would this interact (if it would at all) with mTLS? And how could callers be authenticated?

Programmatic definitions can be added in the future to any Dapr resource, yes.

Authentication could also be supported in the future by letting the user use a cert/cert-key combination same as the HTTP binding does today, and potentially extend it to using the workload cert in case the remote endpoint uses the same trust chain as the calling sidecar instance.

[sicoyle]

One comment/question I have is whether it'd be possible to offer a programmatic way to define these external endpoints too, in addition to using a CRD (like we do for pubsub subscriptions for example).

Also, how would this interact (if it would at all) with mTLS? And how could callers be authenticated?

I can look into a programmatic way of defining external endpoints for the future. Thank you for bringing that up!

For the first iteration, I'll implement the feature with the most basic authentication measures, but can iterate to add more in. It might be interesting for the future to add in support to use the Dapr mTLS certificate.

[ItalyPaleAle]

What do you mean with "the most basic authentication measures"?

[sicoyle]

What do you mean with "the most basic authentication measures"?

API tokens, username/password, etc.

[ItalyPaleAle]

@sicoyle could you please provide an example of how that will work? Authentication is a critical piece of the security story and since we don’t have mTLS like within the Dapr cluster, this should be made clear in the proposal.

[yaron2]

@sicoyle could you please provide an example of how that will work? Authentication is a critical piece of the security story and since we don’t have mTLS like within the Dapr cluster, this should be made clear in the proposal.

Disagree, this is a future implementation detail that should not be made in this initial proposal.

[ItalyPaleAle]

I would argue it is, because there are certain design decisions that are very critical to security:

• Username/password or auth token only are shared secrets so we would need to define how they can be rotated, perhaps with 0-downtime? (I.e. you Can’t restart all apps so they pick up the new secret)
• Should TLS/HTTPS be enforced? Otherwise the secrets are transferred in plain-text.
• Should Dapr generate JWTs to authenticate calls? This is significantly better than shared tokens because they are time- and audience-bound.

[yaron2]

In general this is about talking to endpoints that have their own security postures which are out of our control, essentially just like the HTTP binding and components in general which don't necessarily include authentication at all or to begin with.

In this case this is very similar to the HTTP binding and components in general - if the external non-Dapr endpoint has authentication mechanisms that the external invocation doesn't support, then it simply will not be usable. The strictness requirement is on the external endpoint and it's up to Dapr to provide users with the most options to meet it, but that 100% doesn't need to be tackled from the very beginning and can be incremental just like many other authentication mechanisms we have in different places.

Now regardless, here's what I'm thinking about the issues you raised:

Username/password or auth token only are shared secrets so we would need to define how they can be rotated, perhaps with 0-downtime? (I.e. you Can’t restart all apps so they pick up the new secret)

Restarting multiple replicas in a rolling fashion can provide zero downtime, and this is exactly what happens today for all of our components when a secret, password or token changes. It might be beneficial to consider a general purpose secret rotation mechanism for Dapr sidecars but this is definitely outside the scope of this proposal

Should TLS/HTTPS be enforced? Otherwise the secrets are transferred in plain-text

No, the user should use this feature the same as they would an HTTP client talking to an external endpoint. We can provide a behavior that enforces TLS/HTTPS is an authentication secret is provided.

Should Dapr generate JWTs to authenticate calls? This is significantly better than shared tokens because they are time- and audience-bound

Adding a JWT authentication feature would be good to add in the future.

[ItalyPaleAle]

In general this is about talking to endpoints that have their own security postures which are out of our control, essentially just like the HTTP binding and components in general which don't necessarily include authentication at all or to begin with.

Ok, this is good context-setting. With this premise, I agree that we can push more of the responsibility of authenticating calls towards the user. Let's just make sure this is documented :)

Restarting multiple replicas in a rolling fashion can provide zero downtime, and this is exactly what happens today for all of our components when a secret, password or token changes. It might be beneficial to consider a general purpose secret rotation mechanism for Dapr sidecars but this is definitely outside the scope of this proposal

The problem is coordination because during the rollout you'll have some apps making calls with the old API token, and some with the new one. However, given the point above, I guess this is not our problem :)

No, the user should use this feature the same as they would an HTTP client talking to an external endpoint. We can provide a behavior that enforces TLS/HTTPS is an authentication secret is provided.

Given the premise above, I think this is not necessary.

Adding a JWT authentication feature would be good to add in the future.

Agree. @sicoyle would you mind opening an issue in dapr/dapr so we don't forget about this? :)

[lloydfischer]

I’m one of the instigators of the original ask for this feature. The primary use case for us was to allow a DAPR service to call out to an external service without having to get a token. Ideally, middleware would be invoked on the way out that could get a token and add it to the request. Is that possible under this proposal?

[ItalyPaleAle]

I’m one of the instigators of the original ask for this feature. The primary use case for us was to allow a DAPR service to call out to an external service without having to get a token. Ideally, middleware would be invoked on the way out that could get a token and add it to the request. Is that possible under this proposal?

I would say no.

We don't have a middleware to generate signed tokens (JWTs).

I don't think we could implement this with a middleware either because the JWT provider would also need to expose an endpoint from where to fetch the JWKS with the keys to verify the tokens.

mTLS is another option, although for security reasons we would probably never allow an external app to fetch a certificate from the Dapr Sentry, and we'd need to rely on external PKIs.

Using a JWT is IMHO significantly simpler than meddling with PKIs

[lloydfischer]

Middleware to get tokens already exists. The question is if the call to a non-DAPR endpoint passes through middleware or notOn Apr 11, 2023, at 9:56 PM, Alessandro (Ale) Segala @.***> wrote:

I’m one of the instigators of the original ask for this feature. The primary use case for us was to allow a DAPR service to call out to an external service without having to get a token. Ideally, middleware would be invoked on the way out that could get a token and add it to the request. Is that possible under this proposal?

I would say no. We don't have a middleware to generate signed tokens (JWTs). I don't think we could implement this with a middleware either because the JWT provider would also need to expose an endpoint from where to fetch the JWKS with the keys to verify the tokens. mTLS is another option, although for security reasons we would probably never allow an external app to fetch a certificate from the Dapr Sentry, and we'd need to rely on external PKIs.

Using a JWT is IMHO significantly simpler than meddling with PKIs

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[ItalyPaleAle]

Although there are middlewares to get tokens, they require an external IdP to sign them. Is that a problem?

[yaron2]

The question is if the call to a non-DAPR endpoint passes through middleware or not

Yes, these calls will pass through the middleware.

[lloydfischer]

Thanks @yaron2

@ItalyPaleAle

Although there are middlewares to get tokens, they require an external IdP to sign them. Is that a problem?

Not a problem, that is expected. The middleware should run a client credentials flow to a config specified IDP to get and cache an access token.. Then the outbound call gets the token attached thus relieving the business logic code of the whole problem. Each external endpoint would need its own config since it is in control of the token requirements and will likely specify the IDP to be used.

I expect once we have the basic plumbing in place and the outbound is flowing through the MW we may need a new MW implementation to fill the gap. That will be relatively easy and isolated to the MW. Maybe someday there will be a authentication building block but that is not critical in my opinion. In fact, I'd say that an Authorization BB is more important. But that is all for later.