search

hashicorp / consul-k8s

First-class support for Consul Service Mesh on Kubernetes
https://www.consul.io/docs/k8s
Mozilla Public License 2.0
669 stars 322 forks source link

Delete services synced to external Consul server when running helm delete #156

Open jasonliu747 opened 5 years ago

jasonliu747 commented 5 years ago

Edited by @lkysow: After investigation, this request is about deleting services synced from Kubernetes to an external Consul server when the Consul helm release is deleted. This only applies for external Consul servers because if they were being deployed on Kubernetes, they would be deleted when helm delete is run.

~The service synced to consul by annotation won't deregister after the service is deleted. And won't deregister after changing its annotation to consul.hashicorp.com/service-sync: "false" either.~

~Here is my values.yaml image~

lkysow commented 5 years ago

Hi @jasonliu747 I think this was fixed by https://github.com/hashicorp/consul-k8s/pull/153 however we haven't got a release out yet. We'll get on that ASAP.

(possible duplicate of https://github.com/hashicorp/consul-k8s/issues/76)

jasonliu747 commented 5 years ago

FYI, if the helm release was deleted, services didn't deregister either. Cheers!

lkysow commented 5 years ago

@jasonliu747 can you test with the latest version (v0.9.4) please. I'm not sure what you mean by deleting the helm release? Do you mean Consul or a service you're syncing into Consul that is also managed by helm?

jasonliu747 commented 5 years ago

For example, I have three local VMs as consul server. And deploy consul clients(server disabled in values.yaml) in K8S using helm. Latest version did fix #76. But when I executed helm delete --purge consul, services synced from K8S didn't deregister automatically, and a node called k8s-sync was still in the Nodes page.

lkysow commented 5 years ago

Okay I understand. That's expected behaviour. When the sync catalog exits, it doesn't delete services. If it did, then when you did a deploy of it or its Pod died, your services would be deleted.

jasonliu747 commented 5 years ago

But when someone deletes consul from K8S using helm delete --purge consul, user might expect all the services/nodes synced from K8S should be deleted as well.

lkysow commented 5 years ago

Yes, they might expect that. We could build it using a helm hook I suppose. If that's what you're asking for I'll reword this issue.

jasonliu747 commented 5 years ago

Awesome buddy. Thanks for ur hard working.

till commented 2 years ago

@lkysow is there a manual way to clear services registered through this or is my best bet to delete consul and try again?

lkysow commented 2 years ago

Hi, you can use the Consul HTTP API to delete services.

till commented 2 years ago

@lkysow that doesn't seem to work. A quick google told me that I have to fetch a service ID in order to unregister the service. At least that is what I am hoping you suggested. So I tried it on a test setup, I had previously "synced" the haproxy-ingress-metrics-ingress-controller using consul (or this chart).

cURL: http://localhost:57947/v1/catalog/service/haproxy-ingress-metrics-ingress-controller

[
  {
    "ID": "",
    "Node": "k8s-sync",
    "Address": "127.0.0.1",
    "Datacenter": "testing",
    "TaggedAddresses": null,
    "NodeMeta": {
      "external-source": "kubernetes"
    },
    "ServiceKind": "",
    "ServiceID": "haproxy-ingress-metrics-ingress-controller-4a7b801e5ecc",
    "ServiceName": "haproxy-ingress-metrics-ingress-controller",
    "ServiceTags": [
      "k8s"
    ],
    "ServiceAddress": "10.244.196.215",
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {
      "external-k8s-ns": "ingress-controller",
      "external-source": "kubernetes",
      "port-ctrl-metrics": "10254",
      "port-metrics": "9101"
    },
    "ServicePort": 10254,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
      "Mode": "",
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "CreateIndex": 8006511,
    "ModifyIndex": 8006511
  },
  {
    "ID": "",
    "Node": "k8s-sync",
    "Address": "127.0.0.1",
    "Datacenter": "testing",
    "TaggedAddresses": null,
    "NodeMeta": {
      "external-source": "kubernetes"
    },
    "ServiceKind": "",
    "ServiceID": "haproxy-ingress-metrics-ingress-controller-775a174389f8",
    "ServiceName": "haproxy-ingress-metrics-ingress-controller",
    "ServiceTags": [
      "k8s"
    ],
    "ServiceAddress": "10.244.36.138",
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {
      "external-k8s-ns": "ingress-controller",
      "external-source": "kubernetes",
      "port-ctrl-metrics": "10254",
      "port-metrics": "9101"
    },
    "ServicePort": 10254,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
      "Mode": "",
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "CreateIndex": 8006522,
    "ModifyIndex": 8006522
  },
  {
    "ID": "",
    "Node": "k8s-sync",
    "Address": "127.0.0.1",
    "Datacenter": "testing",
    "TaggedAddresses": null,
    "NodeMeta": {
      "external-source": "kubernetes"
    },
    "ServiceKind": "",
    "ServiceID": "haproxy-ingress-metrics-ingress-controller-fff5b8d95794",
    "ServiceName": "haproxy-ingress-metrics-ingress-controller",
    "ServiceTags": [
      "k8s"
    ],
    "ServiceAddress": "10.244.147.40",
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {
      "external-k8s-ns": "ingress-controller",
      "external-source": "kubernetes",
      "port-ctrl-metrics": "10254",
      "port-metrics": "9101"
    },
    "ServicePort": 10254,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
      "Mode": "",
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "CreateIndex": 8006514,
    "ModifyIndex": 8006514
  }
]

When I use either of the ServiceID I get:

curl --request PUT http://localhost:57947/v1/agent/service/deregister/haproxy-ingress-metrics-ingress-controller-fff5b8d95794
Unknown service ID "haproxy-ingress-metrics-ingress-controller-fff5b8d95794". Ensure that the service ID is passed, not the service name.

Any clue which API to use to clean this up?

lkysow commented 2 years ago

Hey I think you need to use the catalog deregister endpoints. Not the agent endpoints. Sorry for not being exact, I'm not at my laptop.

till commented 2 years ago

Hey I think you need to use the catalog deregister endpoints. Not the agent endpoints. Sorry for not being exact, I'm not at my laptop.

Thank you, much appreciated! :)

So for the record, or for anyone looking to delete all of what the catalog sync added:

curl --request PUT -d '{"Node":"k8s-sync"}' http://consul/v1/catalog/deregister