HTTPRoute resource not working and doesn't show in "kubectl get HTTPRoute"

[bwmetcalf]

What is the issue?

We are using HTTPRoute with MeshTLSAuthentication and upstream requests are failing with 403 regardless of endpoint. This works fine when we just use Server in our authentication policy. Below are all of the manifests. The strange thing here is in order to retrieve our HTTPRoute resource we have to use

kubectl get httproute.policy.linkerd.io

Eg,

~  % k get httproute
No resources found in contact-automation namespace.
~  % k get httproute.policy.linkerd.io
NAME                      HOSTNAMES   AGE
c3-test-service-default               19m
~  % k get server
NAME              PORT   PROTOCOL   ACCESS POLICY
c3-test-service   8080   HTTP/2     deny

Here are all of the manifests:

~  % k get server c3-test-service -oyaml
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"policy.linkerd.io/v1beta3","kind":"Server","metadata":{"annotations":{},"labels":{"app":"c3-test-service","app.kubernetes.io/instance":"c3-test-service-test"},"name":"c3-test-service","namespace":"contact-automation"},"spec":{"podSelector":{"matchLabels":{"app":"c3-test-service"}},"port":8080,"proxyProtocol":"HTTP/2"}}
  creationTimestamp: "2024-10-04T18:19:00Z"
  generation: 44
  labels:
    app: c3-test-service
    app.kubernetes.io/instance: c3-test-service-test
  name: c3-test-service
  namespace: contact-automation
  resourceVersion: "1086082025"
  uid: 30bbb581-7ca5-4d6a-81ec-67a9b2d77ad3
spec:
  accessPolicy: deny
  podSelector:
    matchLabels:
      app: c3-test-service
  port: 8080
  proxyProtocol: HTTP/2

~  % k get httproute.policy.linkerd.io c3-test-service-default -oyaml
apiVersion: policy.linkerd.io/v1beta3
kind: HTTPRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"policy.linkerd.io/v1beta3","kind":"HTTPRoute","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"c3-test-service-test"},"name":"c3-test-service-default","namespace":"contact-automation"},"spec":{"parentRefs":[{"group":"policy.linkerd.io","kind":"Server","name":"c3-test-service"}],"rules":[{"timeouts":{"request":"0"}}]}}
  creationTimestamp: "2024-10-10T18:07:52Z"
  generation: 3
  labels:
    app.kubernetes.io/instance: c3-test-service-test
  name: c3-test-service-default
  namespace: contact-automation
  resourceVersion: "1086082027"
  uid: 2288f7ad-8ec0-44aa-82ec-10a0f4f5d823
spec:
  parentRefs:
  - group: policy.linkerd.io
    kind: Server
    name: c3-test-service
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    timeouts:
      request: "0"
status:
  parents:
  - conditions:
    - lastTransitionTime: "2024-10-10T18:07:52Z"
      message: ""
      reason: Accepted
      status: "True"
      type: Accepted
    controllerName: linkerd.io/policy-controller
    parentRef:
      group: policy.linkerd.io
      kind: Server
      name: c3-test-service
      namespace: contact-automation

~  % k get meshtlsauthentication c3-test-service -oyaml
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"policy.linkerd.io/v1alpha1","kind":"MeshTLSAuthentication","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"c3-test-service-test"},"name":"c3-test-service","namespace":"contact-automation"},"spec":{"identityRefs":[{"kind":"ServiceAccount","name":"c3-test-client","namespace":"contact-automation"}]}}
  creationTimestamp: "2024-10-04T16:42:16Z"
  generation: 2
  labels:
    app.kubernetes.io/instance: c3-test-service-test
  name: c3-test-service
  namespace: contact-automation
  resourceVersion: "1085957436"
  uid: 36022baa-28f1-4ffc-b3aa-518c4466c954
spec:
  identityRefs:
  - kind: ServiceAccount
    name: c3-test-client
    namespace: contact-automation

~  % k get authorizationpolicy c3-test-service-default-c3-test-client -oyaml
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"policy.linkerd.io/v1alpha1","kind":"AuthorizationPolicy","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"c3-test-service-test"},"name":"c3-test-service-default-c3-test-client","namespace":"contact-automation"},"spec":{"requiredAuthenticationRefs":[{"group":"policy.linkerd.io","kind":"MeshTLSAuthentication","name":"c3-test-service-c3-test-client"}],"targetRef":{"group":"policy.linkerd.io","kind":"HTTPRoute","name":"c3-test-service-default"}}}
  creationTimestamp: "2024-10-10T16:10:03Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: c3-test-service-test
  name: c3-test-service-default-c3-test-client
  namespace: contact-automation
  resourceVersion: "1085957450"
  uid: 48f8a122-5bbd-419f-a112-bca41d5b0441
spec:
  requiredAuthenticationRefs:
  - group: policy.linkerd.io
    kind: MeshTLSAuthentication
    name: c3-test-service-c3-test-client
  targetRef:
    group: policy.linkerd.io
    kind: HTTPRoute
    name: c3-test-service-default

How can it be reproduced?

I suppose configure the above resources with our versions of linkerd shown in the check output.

Logs, error output, etc

~  % k exec -it -c app c3-test-client-5f8589549d-lsw56 -- curl -I http://c3-test-service.contact-automation
HTTP/1.1 403 Forbidden
content-length: 0
date: Thu, 10 Oct 2024 18:38:30 GMT

output of linkerd check -o short

~  % linkerd check -o short
linkerd-version
---------------
‼ cli is up-to-date
    unsupported version channel: stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane and cli versions match
    control plane running edge-24.10.2 but cli running stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies and cli versions match
    linkerd-destination-584d85975-545wz running edge-24.10.2 but cli running stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-cp-proxy-cli-version for hints

linkerd-ha-checks
-----------------
‼ pod injection disabled on kube-system
    kube-system namespace needs to have the label config.linkerd.io/admission-webhooks: disabled if injector webhook failure policy is Fail
    see https://linkerd.io/2.14/checks/#l5d-injection-disabled for hints

linkerd-viz
-----------
‼ viz extension proxies and cli versions match
    metrics-api-86dd7f89dd-n6dvp running edge-24.10.2 but cli running stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cli-version for hints

Status check results are √

Environment

% kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.8-eks-a737599

We are using bottlerocket AMIs.

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

None

[bwmetcalf]

Our linkerd-proxy logs show

[ 10161.537196s]  INFO ThreadId(01) inbound:server{port=8080}:rescue{client.addr=10.3.148.114:54096}: linkerd_app_core::errors::respond: HTTP/2.0 request failed error=client 10.3.148.114:54096: server: 10.3.128.129:8080: unauthorized request on route error.sources=[unauthorized request on route]

Perhaps our HTTPRoute resource spec is wrong, but based on the docs it looks right.

[bwmetcalf]

We had a typo in the auth policy. Closing.

[wmorgan]

@bwmetcalf No worries. What was the typo in the auth policy? We are looking into how to improve the usability of HTTPRoutes in Linkerd and this class of error is still an opportunity to improve the UX IMO

[bwmetcalf]

I was referencing the MeshTLSAutentication resource in the auth policy with

  requiredAuthenticationRefs:
  - group: policy.linkerd.io
    kind: MeshTLSAuthentication
    name: c3-test-service-c3-test-client

when the name of the resource is actually c3-test-service. It was an error I introduced in our yaml templating.