search

hashicorp / consul-api-gateway

The Consul API Gateway is a dedicated ingress solution for intelligently routing traffic to applications running on a Consul Service Mesh.
Mozilla Public License 2.0
99 stars 17 forks source link

API-Gateway-Controller fails to resolve Vault Service registered in Consul #299

Open tyler-wendys opened 2 years ago

tyler-wendys commented 2 years ago

Overview of the Issue

The TL;DR: The api-gateway-controller fails to resolve the consul service registration on deployment of httproutes for vault when deploying Vault with a consul HA backend.

The longer version: I have a GKE cluster that I’m running Vault with Consul serving as the HA backend. I’ve installed Vault and Consul via helm charts, everything between those two installations appears to be playing nice. I’m now attempting to set up Consul API Gateway with that Consul cluster to set up some (internal) ingress traffic for a vault-injector living in another cluster. Everything is working together flawlessly until it comes to setting up the HTTPRoute, when doing so the api-gateway-controller fails to resolve the the consul service registration for Vault.

Reproduction Steps

Here's all of the things we've got setup for the installations of Vault + Consul + API Gateway:

Consul helm overrides:

global:
  name: consul
  gossipEncryption: 
    autoGenerate: true
  tls:
    enabled: true
    enableAutoEncrypt: true
    verify: true
    httpsOnly: false
  acls:
    enabled: true
    manageSystemACLs: true
    default_policy: "allow"
    enable_token_persistence: true
ui:
  enabled: true
  type: "LoadBalancer"
client:
  enabled: true
connectInject:
  enabled: true
controller:
  enabled: true
terminatingGateways:
  enabled: false
ingressGateways:
  enabled: false
apiGateway:
  enabled: true
  image: hashicorp/consul-api-gateway:0.3.0
  managedGatewayClass:
    copyAnnotations:
      service: 
        annotations: |
          - 'networking.gke.io/load-balancer-type'

Vault helm overrides:

global:
  enabled: true
  tlsDisable: true
injector:
  enabled: false
  logLevel: debug
  webhook:
    failurePolicy: Fail
  image:
    repository: hashicorp/vault-k8s
    tag: latest
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 256Mi
      cpu: 250m
server:
  resources:
    requests:
      memory: 8Gi
      cpu: 2000m
    limits:
      memory: 16Gi
      cpu: 2000m
  readinessProbe:
    enabled: true
    path: /v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204
  livenessProbe:
    enabled: true
    path: /v1/sys/health?standbyok=true
    initialDelaySeconds: 60
  auditStorage:
    enabled: true
  standalone:
    enabled: false
  ha:
    enabled: true
    replicas: 5
    config: |

      ui = true

      listener "tcp" {
        tls_disable = true
        address = "[::]:8200"
        cluster_address = "[::]:8201"    
      }

      storage "consul" {
          path = "vault/"
          address = "<consul_addr>"
          token = "<consul_token>"
          scheme = "https"
          tls_skip_verify = true
      }

API Gateway resources: (I'm including the TLS block in case it turns out to be a clue, but I have validated that the api gateway at least receives traffic ok over ssl)

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  name: vault-gateway
  namespace: vault
  annotations:
    networking.gke.io/load-balancer-type: 'Internal'
spec:
  gatewayClassName: consul-api-gateway
  listeners:
    - protocol: HTTPS
      hostname: internal.vault.hostname
      port: 443
      name: https
      allowedRoutes:
        namespaces:
          from: Same
      tls:
        certificateRefs:
          - name: vault-ingress-certificate
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: vault-route
  namespace: vault
spec:
  parentRefs:
    - name: vault-gateway
  rules:
    - backendRefs:
        - kind: Service
          name: vault
          namespace: vault
          port: 8200
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: ReferencePolicy
metadata:
  name: vault-ref-policy-route
  namespace: vault
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: vault
  to:
    - group: ""
      kind: Service
      name: vault

Logs

2022-08-04T20:42:28.917Z [ERROR] service/resolver.go:249: consul-api-gateway-server.k8s.Reconciler: could not resolve consul service: error="consul service vault/vault not found"
2022-08-04T20:43:00.370Z [ERROR] service/resolver.go:249: consul-api-gateway-server.k8s.Reconciler: could not resolve consul service: error="consul service vault/vault not found"
2022-08-04T20:43:31.315Z [ERROR] service/resolver.go:249: consul-api-gateway-server.k8s.Reconciler: could not resolve consul service: error="consul service vault/vault not found"

The HTTPRoute registration

apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1alpha2
  kind: HTTPRoute
  metadata:
    name: vault-route
    namespace: vault
  spec:
    parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: vault-gateway
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: vault
        port: 8200
        weight: 1
      matches:
      - path:
          type: PathPrefix
          value: /
  status:
    parents:
    - conditions:
      - lastTransitionTime: "2022-08-04T20:42:28Z"
        message: Route accepted.
        observedGeneration: 3
        reason: Accepted
        status: "True"
        type: Accepted
      - lastTransitionTime: "2022-08-04T20:42:28Z"
        message: 'consul: consul service vault/vault not found'
        observedGeneration: 3
        reason: ConsulServiceNotFound
        status: "False"
        type: ResolvedRefs
      controllerName: hashicorp.com/consul-api-gateway-controller
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: vault-gateway
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Expected behavior

The HTTPRoute registers both the kubernetes and consul service registrations for vault and allows for the api gateway to route traffic to vault.

Environment details

Additional Context

In case this helps, below is also a listing of the vault service registered in consul.

{
  "vault:10.4.0.59:8200": {
    "ID": "vault:10.4.0.59:8200",
    "Service": "vault",
    "Tags": [
      "standby",
      "initialized"
    ],
    "Meta": {
      "external-source": "vault"
    },
    "Port": 8200,
    "Address": "10.4.0.59",
    "TaggedAddresses": {
      "lan_ipv4": {
        "Address": "10.4.0.59",
        "Port": 8200
      },
      "wan_ipv4": {
        "Address": "10.4.0.59",
        "Port": 8200
      }
    },
    "Weights": {
      "Passing": 1,
      "Warning": 1
    },
    "EnableTagOverride": false,
    "Datacenter": "dc1"
  },
  "vault:10.4.1.89:8200": {
    "ID": "vault:10.4.1.89:8200",
    "Service": "vault",
    "Tags": [
      "active",
      "initialized"
    ],
    "Meta": {
      "external-source": "vault"
    },
    "Port": 8200,
    "Address": "10.4.1.89",
    "TaggedAddresses": {
      "lan_ipv4": {
        "Address": "10.4.1.89",
        "Port": 8200
      },
      "wan_ipv4": {
        "Address": "10.4.1.89",
        "Port": 8200
      }
    },
    "Weights": {
      "Passing": 1,
      "Warning": 1
    },
    "EnableTagOverride": false,
    "Datacenter": "dc1"
  },
  "vault:10.4.3.5:8200": {
    "ID": "vault:10.4.3.5:8200",
    "Service": "vault",
    "Tags": [
      "standby",
      "initialized"
    ],
    "Meta": {
      "external-source": "vault"
    },
    "Port": 8200,
    "Address": "10.4.3.5",
    "TaggedAddresses": {
      "lan_ipv4": {
        "Address": "10.4.3.5",
        "Port": 8200
      },
      "wan_ipv4": {
        "Address": "10.4.3.5",
        "Port": 8200
      }
    },
    "Weights": {
      "Passing": 1,
      "Warning": 1
    },
    "EnableTagOverride": false,
    "Datacenter": "dc1"
  }
}

Thank you for taking a look at this!

mikemorris commented 2 years ago

It looks like the error you're hitting is https://github.com/hashicorp/consul-api-gateway/blob/v0.3.0/internal/k8s/service/resolver.go#L244 - could you try capturing Consul API Gateway controller logs with logLevel: trace configured in the Helm chart? (trace doesn't appear to be documented in the Helm config, but should pass through to set https://pkg.go.dev/github.com/hashicorp/go-hclog#Level) which will hopefully offer more detail as to where specifically this lookup is failing.

The code path you're getting an error from iterates over services registered to all Consul nodes - what I'm not quite sure about is how the Vault backend integration registers itself with Consul - it may be as an external service which could be an edge case our logic doesn't cover yet.

tyler-wendys commented 2 years ago

Thank you so much for getting back to me! We tried bumping the logLevel to trace, resulting in this message back from the controller:

2022-08-08T19:30:03.455Z [TRACE] reconciler/route.go:251: consul-api-gateway-server.k8s.Reconciler.route: syncing route status: name=vault-route
  status=
  | {
  |   "parents": [
  |     {
  |       "parentRef": {
  |         "group": "gateway.networking.k8s.io",
  |         "kind": "Gateway",
  |         "name": "vault-gateway"
  |       },
  |       "controllerName": "hashicorp.com/consul-api-gateway-controller",
  |       "conditions": [
  |         {
  |           "type": "Accepted",
  |           "status": "True",
  |           "observedGeneration": 4,
  |           "lastTransitionTime": "2022-08-08T19:30:03Z",
  |           "reason": "Accepted",
  |           "message": "Route accepted."
  |         },
  |         {
  |           "type": "ResolvedRefs",
  |           "status": "False",
  |           "observedGeneration": 4,
  |           "lastTransitionTime": "2022-08-08T19:30:03Z",
  |           "reason": "ConsulServiceNotFound",
  |           "message": "consul: consul service vault/vault not found"
  |         }
  |       ]
  |     }
  |   ]
  | }

2022-08-08T19:30:03.465Z [TRACE] reconciler/gateway.go:523: consul-api-gateway-server.k8s.Reconciler.gateway: created or updated gateway service: name=vault-gateway namespace=vault
  service=
  | {
  |   "kind": "Service",
  |   "apiVersion": "v1",
  |   "metadata": {
  |     "name": "vault-gateway",
  |     "namespace": "vault",
  |     "uid": "5d9dd04e-a5a0-4778-a69b-5a0cec17e0b2",
  |     "resourceVersion": "846248",
  |     "creationTimestamp": "2022-08-04T20:42:28Z",
  |     "labels": {
  |       "api-gateway.consul.hashicorp.com/created": "1659645717",
  |       "api-gateway.consul.hashicorp.com/managed": "true",
  |       "api-gateway.consul.hashicorp.com/name": "vault-gateway",
  |       "api-gateway.consul.hashicorp.com/namespace": "vault"
  |     },
  |     "annotations": {
  |       "cloud.google.com/neg": "{\"ingress\":true}",
  |       "external-dns.alpha.kubernetes.io/hostname": "example.hostname",
  |       "networking.gke.io/load-balancer-type": "Internal"
  |     },
  |     "ownerReferences": [
  |       {
  |         "apiVersion": "gateway.networking.k8s.io/v1alpha2",
  |         "kind": "Gateway",
  |         "name": "vault-gateway",
  |         "uid": "0b9a4e07-836d-41e8-b5bf-58301dfec862",
  |         "controller": true,
  |         "blockOwnerDeletion": true
  |       }
  |     ],
  |     "finalizers": [
  |       "gke.networking.io/l4-ilb-v1",
  |       "service.kubernetes.io/load-balancer-cleanup"
  |     ],
  |     "managedFields": [
  |       {
  |         "manager": "consul-api-gateway",
  |         "operation": "Update",
  |         "apiVersion": "v1",
  |         "time": "2022-08-04T20:42:28Z",
  |         "fieldsType": "FieldsV1",
  |         "fieldsV1": {
  |           "f:metadata": {
  |             "f:annotations": {
  |               ".": {},
  |               "f:external-dns.alpha.kubernetes.io/hostname": {},
  |               "f:networking.gke.io/load-balancer-type": {}
  |             },
  |             "f:labels": {
  |               ".": {},
  |               "f:api-gateway.consul.hashicorp.com/created": {},
  |               "f:api-gateway.consul.hashicorp.com/managed": {},
  |               "f:api-gateway.consul.hashicorp.com/name": {},
  |               "f:api-gateway.consul.hashicorp.com/namespace": {}
  |             },
  |             "f:ownerReferences": {
  |               ".": {},
  |               "k:{\"uid\":\"0b9a4e07-836d-41e8-b5bf-58301dfec862\"}": {
  |                 ".": {},
  |                 "f:apiVersion": {},
  |                 "f:blockOwnerDeletion": {},
  |                 "f:controller": {},
  |                 "f:kind": {},
  |                 "f:name": {},
  |                 "f:uid": {}
  |               }
  |             }
  |           },
  |           "f:spec": {
  |             "f:externalTrafficPolicy": {},
  |             "f:ports": {
  |               ".": {},
  |               "k:{\"port\":443,\"protocol\":\"TCP\"}": {
  |                 ".": {},
  |                 "f:name": {},
  |                 "f:port": {},
  |                 "f:protocol": {},
  |                 "f:targetPort": {}
  |               }
  |             },
  |             "f:selector": {
  |               ".": {},
  |               "f:api-gateway.consul.hashicorp.com/created": {},
  |               "f:api-gateway.consul.hashicorp.com/managed": {},
  |               "f:api-gateway.consul.hashicorp.com/name": {},
  |               "f:api-gateway.consul.hashicorp.com/namespace": {}
  |             },
  |             "f:sessionAffinity": {},
  |             "f:type": {}
  |           }
  |         }
  |       },
  |       {
  |         "manager": "kube-controller-manager",
  |         "operation": "Update",
  |         "apiVersion": "v1",
  |         "time": "2022-08-04T20:43:06Z",
  |         "fieldsType": "FieldsV1",
  |         "fieldsV1": {
  |           "f:metadata": {
  |             "f:finalizers": {
  |               ".": {},
  |               "v:\"gke.networking.io/l4-ilb-v1\"": {},
  |               "v:\"service.kubernetes.io/load-balancer-cleanup\"": {}
  |             }
  |           },
  |           "f:status": {
  |             "f:loadBalancer": {
  |               "f:ingress": {}
  |             }
  |           }
  |         }
  |       }
  |     ]
  |   },
  |   "spec": {
  |     "ports": [
  |       {
  |         "name": "https",
  |         "protocol": "TCP",
  |         "port": 443,
  |         "targetPort": 443,
  |         "nodePort": 32520
  |       }
  |     ],
  |     "selector": {
  |       "api-gateway.consul.hashicorp.com/created": "1659645717",
  |       "api-gateway.consul.hashicorp.com/managed": "true",
  |       "api-gateway.consul.hashicorp.com/name": "vault-gateway",
  |       "api-gateway.consul.hashicorp.com/namespace": "vault"
  |     },
  |     "clusterIP": "10.192.51.251",
  |     "clusterIPs": [
  |       "10.192.51.251"
  |     ],
  |     "type": "LoadBalancer",
  |     "sessionAffinity": "None",
  |     "externalTrafficPolicy": "Cluster",
  |     "ipFamilies": [
  |       "IPv4"
  |     ],
  |     "ipFamilyPolicy": "SingleStack"
  |   },
  |   "status": {
  |     "loadBalancer": {
  |       "ingress": [
  |         {
  |           "ip": "192.2.0.12"
  |         }
  |       ]
  |     }
  |   }
  | }

2022-08-08T19:30:03.467Z [TRACE] memory/store.go:227: consul-api-gateway-server.state: detected gateway state change: service=vault-gateway namespace=""
2022-08-08T19:30:03.467Z [TRACE] memory/gateway.go:53: consul-api-gateway-server.state: checking if route can bind to gateway: gateway.consul.namespace="" gateway.consul.service=vault-gateway route=http-vault/vault-route
2022-08-08T19:30:03.467Z [TRACE] memory/gateway.go:58: consul-api-gateway-server.state: checking if route can bind to listener: gateway.consul.namespace="" gateway.consul.service=vault-gateway listener=https route=http-vault/vault-route
2022-08-08T19:30:03.467Z [TRACE] reconciler/listener.go:336: consul-api-gateway-server.k8s.Reconciler.gateway.listener: checking route parent ref: listener=https name=vault-gateway namespace=vault name=vault-gateway
2022-08-08T19:30:03.467Z [TRACE] reconciler/listener.go:339: consul-api-gateway-server.k8s.Reconciler.gateway.listener: checking gateway match: listener=https name=vault-gateway namespace=vault expected=vault/vault-gateway found=vault/vault-gateway
2022-08-08T19:30:03.467Z [TRACE] reconciler/listener.go:356: consul-api-gateway-server.k8s.Reconciler.gateway.listener: listener not ready, unable to bind: listener=https name=vault-gateway namespace=vault route=http-vault/vault-route
2022-08-08T19:30:03.477Z [TRACE] reconciler/gateway.go:523: consul-api-gateway-server.k8s.Reconciler.gateway: created or updated gateway service: name=vault-gateway namespace=vault
  service=
  | {
  |   "kind": "Service",
  |   "apiVersion": "v1",
  |   "metadata": {
  |     "name": "vault-gateway",
  |     "namespace": "vault",
  |     "uid": "5d9dd04e-a5a0-4778-a69b-5a0cec17e0b2",
  |     "resourceVersion": "846248",
  |     "creationTimestamp": "2022-08-04T20:42:28Z",
  |     "labels": {
  |       "api-gateway.consul.hashicorp.com/created": "1659645717",
  |       "api-gateway.consul.hashicorp.com/managed": "true",
  |       "api-gateway.consul.hashicorp.com/name": "vault-gateway",
  |       "api-gateway.consul.hashicorp.com/namespace": "vault"
  |     },
  |     "annotations": {
  |       "cloud.google.com/neg": "{\"ingress\":true}",
  |       "external-dns.alpha.kubernetes.io/hostname": "example.hostname",
  |       "networking.gke.io/load-balancer-type": "Internal"
  |     },
  |     "ownerReferences": [
  |       {
  |         "apiVersion": "gateway.networking.k8s.io/v1alpha2",
  |         "kind": "Gateway",
  |         "name": "vault-gateway",
  |         "uid": "0b9a4e07-836d-41e8-b5bf-58301dfec862",
  |         "controller": true,
  |         "blockOwnerDeletion": true
  |       }
  |     ],
  |     "finalizers": [
  |       "gke.networking.io/l4-ilb-v1",
  |       "service.kubernetes.io/load-balancer-cleanup"
  |     ],
  |     "managedFields": [
  |       {
  |         "manager": "consul-api-gateway",
  |         "operation": "Update",
  |         "apiVersion": "v1",
  |         "time": "2022-08-04T20:42:28Z",
  |         "fieldsType": "FieldsV1",
  |         "fieldsV1": {
  |           "f:metadata": {
  |             "f:annotations": {
  |               ".": {},
  |               "f:external-dns.alpha.kubernetes.io/hostname": {},
  |               "f:networking.gke.io/load-balancer-type": {}
  |             },
  |             "f:labels": {
  |               ".": {},
  |               "f:api-gateway.consul.hashicorp.com/created": {},
  |               "f:api-gateway.consul.hashicorp.com/managed": {},
  |               "f:api-gateway.consul.hashicorp.com/name": {},
  |               "f:api-gateway.consul.hashicorp.com/namespace": {}
  |             },
  |             "f:ownerReferences": {
  |               ".": {},
  |               "k:{\"uid\":\"0b9a4e07-836d-41e8-b5bf-58301dfec862\"}": {
  |                 ".": {},
  |                 "f:apiVersion": {},
  |                 "f:blockOwnerDeletion": {},
  |                 "f:controller": {},
  |                 "f:kind": {},
  |                 "f:name": {},
  |                 "f:uid": {}
  |               }
  |             }
  |           },
  |           "f:spec": {
  |             "f:externalTrafficPolicy": {},
  |             "f:ports": {
  |               ".": {},
  |               "k:{\"port\":443,\"protocol\":\"TCP\"}": {
  |                 ".": {},
  |                 "f:name": {},
  |                 "f:port": {},
  |                 "f:protocol": {},
  |                 "f:targetPort": {}
  |               }
  |             },
  |             "f:selector": {
  |               ".": {},
  |               "f:api-gateway.consul.hashicorp.com/created": {},
  |               "f:api-gateway.consul.hashicorp.com/managed": {},
  |               "f:api-gateway.consul.hashicorp.com/name": {},
  |               "f:api-gateway.consul.hashicorp.com/namespace": {}
  |             },
  |             "f:sessionAffinity": {},
  |             "f:type": {}
  |           }
  |         }
  |       },
  |       {
  |         "manager": "kube-controller-manager",
  |         "operation": "Update",
  |         "apiVersion": "v1",
  |         "time": "2022-08-04T20:43:06Z",
  |         "fieldsType": "FieldsV1",
  |         "fieldsV1": {
  |           "f:metadata": {
  |             "f:finalizers": {
  |               ".": {},
  |               "v:\"gke.networking.io/l4-ilb-v1\"": {},
  |               "v:\"service.kubernetes.io/load-balancer-cleanup\"": {}
  |             }
  |           },
  |           "f:status": {
  |             "f:loadBalancer": {
  |               "f:ingress": {}
  |             }
  |           }
  |         }
  |       }
  |     ]
  |   },
  |   "spec": {
  |     "ports": [
  |       {
  |         "name": "https",
  |         "protocol": "TCP",
  |         "port": 443,
  |         "targetPort": 443,
  |         "nodePort": 32520
  |       }
  |     ],
  |     "selector": {
  |       "api-gateway.consul.hashicorp.com/created": "1659645717",
  |       "api-gateway.consul.hashicorp.com/managed": "true",
  |       "api-gateway.consul.hashicorp.com/name": "vault-gateway",
  |       "api-gateway.consul.hashicorp.com/namespace": "vault"
  |     },
  |     "clusterIP": "<cluster_ip>",
  |     "clusterIPs": [
  |       "<cluster_ip>"
  |     ],
  |     "type": "LoadBalancer",
  |     "sessionAffinity": "None",
  |     "externalTrafficPolicy": "Cluster",
  |     "ipFamilies": [
  |       "IPv4"
  |     ],
  |     "ipFamilyPolicy": "SingleStack"
  |   },
  |   "status": {
  |     "loadBalancer": {
  |       "ingress": [
  |         {
  |           "ip": "192.2.0.12"
  |         }
  |       ]
  |     }
  |   }
  | }

Definitely more info coming back, but we haven't seen anything to clue us in on whether or not vault is registering itself as an external service. We're going to keep doing some digging on our end as to whether or not that's the case. Out of curiosity what is your timeline for external services support?

mikemorris commented 2 years ago

Hmm, those logs mostly look like the gateway itself spinning up normally, the only thing that looked off is

reconciler/listener.go:356: consul-api-gateway-server.k8s.Reconciler.gateway.listener: listener not ready, unable to bind: listener=https name=vault-gateway namespace=vault route=http-vault/vault-route

but I'm guessing that was a temporary state given that the route eventually gets an Accepted { status: true } condition.

It may be worth checking the ListenerStatus field on the Gateway to make sure everything looks okay first, then checking the controller logs specifically for errors originating from service/resolver.go to see if anything shows up from https://github.com/hashicorp/consul-api-gateway/blob/v0.3.0/internal/k8s/service/resolver.go#L282-L331

(EDIT: Looking more closely, it appears that Vault uses a standard agent.ServiceRegister call to register itself in Consul, so I don't think external services would be relevant for this case.)

nathancoleman commented 1 year ago

I did a little bit of digging here today and now understand more as to why Consul API Gateway's controller isn't able to resolve the reference to the vault Service.

The ServiceMeta that the controller filters on here - specifically k8s-namespace and k8s-service-name - are not present on the Consul service corresponding with the Kubernetes Service named "vault".

For a "normal" Service that I create myself, you can see the metadata referenced above is present:

$ curl https://localhost:8501/v1/catalog/service/echo-1 --insecure --silent | jq
[
  {
    ...
    "ServiceID": "echo-1-bf966d7c9-nt5dh-echo-1",
    "ServiceName": "echo-1",
    "ServiceTags": [],
    "ServiceMeta": {
      "k8s-namespace": "default",
      "k8s-service-name": "echo-1",
      "managed-by": "consul-k8s-endpoints-controller",
      "pod-name": "echo-1-bf966d7c9-nt5dh",
      "synthetic-node": "true"
    },
    "CreateIndex": 2508,
    "ModifyIndex": 2508
  }
]

For the Service created by the vault Helm chart, it is missing:

$ curl https://localhost:8501/v1/catalog/service/vault --insecure --silent | jq
[
  {
    ...
    "ServiceID": "vault:10.60.0.9:8200",
    "ServiceName": "vault",
    "ServiceTags": [
      "standby"
    ],
    "ServiceMeta": {
      "external-source": "vault"
    },
    "CreateIndex": 1737,
    "ModifyIndex": 1737
  }
  ...
]