HTTPRoute Weight Not Honored for Pod-to-Pod Communication via ClusterIP Service

Manikkumar1988 commented 4 weeks ago

What is the issue?

In minikube with latest linkerd installed,

Expectation is HTTPRoute - should honour the weightage when routing the traffic to desired service for the communication happening through ClusterIP type Service.

Actual is HTTPRoute - doesn't honour the weightage when routing the traffic to desired service. Its alway directing to the non-desired service. Even when weightage of desired service has 100% while the other has 0%

How can it be reproduced?

Create a Namespace and Label it for Linkerd Injection

kubectl create namespace test-app
kubectl annotate namespace test-app linkerd.io/inject=enabled

Create two deployments and expose it by service of clusterIP type. sample-app-v1.yaml


apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-app-v1
namespace: test-app
spec:
replicas: 2
selector:
matchLabels:
  app: sample-app
  version: v1
template:
metadata:
  labels:
    app: sample-app
    version: v1
spec:
  containers:
    - name: http-app
      image: hashicorp/http-echo:latest
      args:
        - "-text=Hello from v1"
      ports:
        - containerPort: 5678

apiVersion: v1 kind: Service metadata: name: sample-app-v1 namespace: test-app spec: selector: app: sample-app version: v1 ports:

port: 80 targetPort: 5678

sample-app-v2.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app-v2
  namespace: test-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sample-app
      version: v2
  template:
    metadata:
      labels:
        app: sample-app
        version: v2
    spec:
      containers:
        - name: http-app
          image: hashicorp/http-echo:latest
          args:
            - "-text=Hello from v2"
          ports:
            - containerPort: 5678

---
apiVersion: v1
kind: Service
metadata:
  name: sample-app-v2
  namespace: test-app
spec:
  selector:
    app: sample-app
    version: v2
  ports:
    - port: 80
      targetPort: 5678

Create one HTTPRoute with two backends(v2 service & v1 service) and one parentRef(v1 service) and set value of weight for the v2 service as 100%. route.yaml

apiVersion: policy.linkerd.io/v1beta2
kind: HTTPRoute
metadata:
name: bb-route
namespace: test-app
spec:
parentRefs:
- name: sample-app-v1
  kind: Service
  group: core
  port: 80
rules:
- backendRefs:
  - name: sample-app-v1
    port: 80
    weight: 0
  - name: sample-app-v2
    port: 80
    weight: 100

Run a nginx pod
kubectl run my-pod --image=nginx -n test-app
Run curl request to v1 service. kubectl exec -it -n test-app my-pod -c my-pod -- curl -s -o - sample-app-v1

Expected: Hello from v2

Actual: Hello from v1

Logs, error output, etc

Expected: Hello from v2

Actual: Hello from v1

output of `linkerd check -o short`

Environment

Kubernetes Version: Client Version: v1.30.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.0

Cluster Version: minikube version: v1.33.1 commit: 5883c09216182566a63dff4c326a6fc9ed2982ff

Host OS: Mac Sonoma 14.5

Linkerd Version: Client version: edge-24.10.4 Server version: edge-24.10.4

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

maybe

kflynn commented 3 weeks ago

Hey @Manikkumar1988! Can you show us the status on that HTTPRoute? kubectl get -n test-app httproute bb-route -o yaml? If you're finding that an HTTPRoute or GRPCRoute isn't working, that's always the first thing to check...

Manikkumar1988 commented 3 weeks ago

Thanks for the response, I have mentioned wrong port in the HTTPRoute thats why it wasn't working. However in my actual, dev env cluster with buoyant image, still issue is reproducible.

olix0r commented 1 week ago

I'm unable to reproduce this:

:; kubectl exec -it -n test-app my-pod -c my-pod -- curl -vs sample-app-v1
*   Trying 10.43.101.1:80...
* Connected to sample-app-v1 (10.43.101.1) port 80 (#0)
> GET / HTTP/1.1
> Host: sample-app-v1
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< x-app-name: http-echo
< x-app-version: 1.0.0
< date: Fri, 15 Nov 2024 20:42:07 GMT
< content-length: 14
< content-type: text/plain; charset=utf-8
< 
Hello from v2
* Connection #0 to host sample-app-v1 left intact

As mentioned, if you encounter this in the future, please capture the route status as well as the output of linkerd diagnostics policy -n test-app svc/sample-app-v1 80.

olix0r commented 3 days ago

We believe this issue may was fixed by https://github.com/linkerd/linkerd2/pull/12635, which was included in edge-24.5.4 (released in May).

Your report indicates that you were using a recent edge release, though:

Linkerd Version: Client version: edge-24.10.4 Server version: edge-24.10.4

If you can reproduce this on a recent Linkerd release, we'd be curious to learn more.

linkerd / linkerd2