Open AlexGoris-KasparSolutions opened 1 year ago
Hi @AlexGoris-KasparSolutions. While Linkerd doesn't automatically authorize traffic for grpc probes yet, you should be able to explicitly authorize these probes by creating an AuthorizationPolicy resource for them. We'll look at adding support for automatic authorization of gRPC probes in the future.
@adleong thanks for the suggestion! I read through the docs of the new AuthorizationPolicy
resource, and I may be missing something, but I don't see an obvious way to create a policy that identities and permits the kubelet's probe requests.
More or less the same issue as described in #7050 for HTTP requests, before those were automatically allowed.
I am also on AKS, with Azure CNI networking, meaning that pods and nodes share the same address range, so I can't make an exception based on source ip.
Is there a better way I'm missing? The goal is that no other pods (besides ones allowed specifically through policies) can communicate with this gRPC service, but allow the livenessProbe requests through.
So I was missing something.
The new HTTPRoute
and AuhorizationPolicy
resources allow specifying a POST
request (which all gRPC requests are) to the health check route, then authorizing those requests to anyone. Since I'm not concerned with rogue pods querying the applications health check call, this works for me.
For reference, adding the following on top of what I described in my initial post got things working:
apiVersion: policy.linkerd.io/v1beta1
kind: HTTPRoute
metadata:
name: grpc-greeter-health-check
spec:
parentRefs:
- name: grpc-greeter
kind: Server
group: policy.linkerd.io
rules:
- matches:
- path:
value: "/grpc.health.v1.Health/Check"
method: POST
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
name: grpc-greeter-health-check
spec:
targetRef:
group: policy.linkerd.io
kind: HTTPRoute
name: grpc-greeter-health-check
requiredAuthenticationRefs: []
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.
Hey @adleong can you give me a brief of what is expected out of this issue? Then I can start working on it. Also, it looks like @AlexGoris-KasparSolutions problem was solved.
Hey @adleong can you give me a brief of what is expected out of this issue? Then I can start working on it. Also, it looks like @AlexGoris-KasparSolutions problem was solved.
Well, I found a workaround, I wouldn't say the issue is fixed. As mentioned in my previous post, this workaround opens the pod up to health-check requests from any other pod, which in theory could be exploited by rogue pods. Besides that concern, it adds a considerable amount of configuration to our k8s infrastructure, whereas non-gRPC health checks are detected by linkerd-proxy and allowed automatically (as described in #7050)
Well, I found a workaround, I wouldn't say the issue is fixed. As mentioned in my previous post, this workaround opens the pod up to health-check requests from any other pod, which in theory could be exploited by rogue pods. Besides that concern, it adds a considerable amount of configuration to our k8s infrastructure, whereas non-gRPC health checks are detected by linkerd-proxy and allowed automatically (as described in #7050)
Cool, I will have a look at it then.
What is the issue?
I closely followed #7050 and was happy to see it was solved in 2.12.0, we took the time this week to upgrade our linkerd installation on our dev cluster (AKS using azure CNI, if that matters). Indeed HTTP/1 health checks worked flawlessly out of the box. Unfortunately I couldn't get HTTP/2 gRPC health checks working.
How can it be reproduced?
I have a basic .NET gRPC service (created as documented here) extended with a basic health check service (GrpcGreeter app extended like documented here). Locally I can call the health check service without problems.
I then deploy the docker image of this build into a namespace with has the following annotations:
The
Deployment
resource has the followingspec.containers[0].livenessProbe
configured:Then added following service:
And defined the following
Server
resource for it:Logs, error output, etc
I can see the linkerd proxy is blocking the livenessProbe connections in the proxy's log:
output of
linkerd check -o short
Environment
Possible solution
No response
Additional context
Important to note is that removing the
Server
resource allows the livenessProbe checks to go through.I wanted to mention that my configuration for an http livenessProbe is exactly the same as aforementioned 'how to reproduce' description, except for the service being an ASP.NET Web API with health checks enabled on the
/healthz
route, and the deployment'slivenessProbe
configured like so:Would you like to work on fixing this bug?
No response