How Can I Run Consul Cluster and Spring Cloud Service On Kubernetes

charles1503 commented 4 years ago

Question

I used the Consul Cloud-auto-joining mode to build a three-node Consul cluster, and used SpringCloud microservices to register with the Consul registry.Because I used the Kuberneters Service, these microservices were decentralized and registered to three nodes. When updating the microservices, the deregistration request was loaded by the Kuberneters Service to the non-original registration node.I found Failed to deregister service: service = charles-service-10-244-5-211-8080 error = "in the log Service {"charles-service-10-244-5-211-8080" {}} does not exist ", so how should I configure so that the unregistered request is loaded to the corresponding node?

consul-cluster.yaml

apiVersion: v1
kind: Service
metadata:
  name: consul
  namespace: charles
  labels:
    name: consul
spec:
  # type: NodePort
  # clusterIP: 10.98.72.72
  type: NodePort
  ports:
    - name: http
      port: 8500
      targetPort: 8500
      nodePort: 31820
    - name: https
      port: 8443
      targetPort: 8443
    - name: rpc
      port: 8400
      targetPort: 8400
    - name: serflan-tcp
      protocol: "TCP"
      port: 8301
      targetPort: 8301
    - name: serflan-udp
      protocol: "UDP"
      port: 8301
      targetPort: 8301
    - name: serfwan-tcp
      protocol: "TCP"
      port: 8302
      targetPort: 8302
    - name: serfwan-udp
      protocol: "UDP"
      port: 8302
      targetPort: 8302
    - name: server
      port: 8300
      targetPort: 8300
    - name: consuldns
      port: 8600
      targetPort: 8600
  selector:
    app: consul

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: consul
  namespace: charles
  labels:
    app: consul
rules:
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
      - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: consul
  namespace: charles
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: consul
subjects:
  - kind: ServiceAccount
    name: consul
    namespace: charles
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: consul
  namespace: charles
  labels:
    app: consul
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: consul-config
  namespace: charles
data:
  server.json: |-
    {
    "bind_addr": "0.0.0.0",
    "client_addr": "0.0.0.0",
    "disable_host_node_id": true,
    "data_dir": "/consul/data",
    "datacenter": "dc1",
    "domain": "cluster.local",
    "retry_join": [
      "provider=k8s namespace=charles label_selector=\"app=consul,component=server\""
    ],
    "server": true,
    "telemetry": {
      "prometheus_retention_time": "5m"
    },
    "ui": true
    }

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: consul
  namespace: charles
spec:
  selector:
    matchLabels:
      app: consul
      component: server
  serviceName: consul
  podManagementPolicy: "Parallel"
  replicas: 3
  template:
    metadata:
      labels:
        app: consul
        component: server
      annotations:
        "consul.hashicorp.com/connect-inject": "false"
    spec:
      serviceAccountName: consul
      terminationGracePeriodSeconds: 10
      securityContext:
        fsGroup: 1000
      containers:
        - name: consul
          image: "consul:1.7.2"
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          args:
            - "agent"
            - "-advertise=$(POD_IP)"
            - "-bootstrap-expect=3"
            - "-config-file=/etc/consul/config/server.json"
          volumeMounts:
            - name: consul-config
              mountPath: /etc/consul/config
          lifecycle:
            preStop:
              exec:
                command:
                - /bin/sh
                - -c
                - consul leave
          ports:
            - containerPort: 8500
              name: ui-port
            - containerPort: 8400
              name: alt-port
            - containerPort: 53
              name: udp-port
            - containerPort: 8443
              name: https-port
            - containerPort: 8080
              name: http-port
            - containerPort: 8301
              name: serflan
            - containerPort: 8302
              name: serfwan
            - containerPort: 8600
              name: consuldns
            - containerPort: 8300
              name: server
      volumes:
        - name: consul-config
          configMap:
            name: consul-config

Problem Desc

a micro service ip 10.244.5.211,port 8080
registered on consul-2

when micro service shutdown ,deregister request to consul-1 , consul-1 log

Failed to deregister service: service = charles-service-10-244-5-211-8080 error = "in the log Service {"charles-service-10-244-5-211-8080" {}} does not exist "

found charles-service-10-244-5-211-8080 health fail on then web ui,Actually,this service has stoped
Solution

use kuberneters lifecycle prestop exec command:

lifecycle:
preStop:
exec:
   command:
   - "/bin/sh"
   - "-c"
   - "curl -X PUT http://consul-0.consul.${NAMESPACE}.svc.cluster.local:8500/v1/agent/service/deregister/charles-service-${POD_IP//./-}-8080"
   - "&& curl -X PUT http://consul-1.consul.${NAMESPACE}.svc.cluster.local:8500/v1/agent/service/deregister/charles-service-${POD_IP//./-}-8080"
   - "&& curl -X PUT http://consul-2.consul.${NAMESPACE}.svc.cluster.local:8500/v1/agent/service/deregister/charles-service-${POD_IP//./-}-8080"

Run a consul client in deployment,use consul client sync consul cluster data,micro-service register consul use "localhost"

containers:
- name: charles-service
...
- name: charles-service-consul-client
  image: consul:1.7.2
  env:
    - name: POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
  args:
    - "agent"
    - "-node=consul-node-$(POD_IP)"
    - "-client"
    - "0.0.0.0"
    - "-join=consul"
  ports:
    - containerPort: 8500

Is there a better way, or am I using it incorrectly?

david-yu commented 4 years ago

Hi @charles1503! It looks like you're trying to run Consul on Kubernetes, we actually do have a Consul for Kubernetes distribution available here: https://www.consul.io/docs/platform/k8s/run.html. I'm wondering if you were able to try it out or if that was perhaps not an option for you? Thanks!

david-yu commented 4 years ago

Closing this issue for now. Please respond if you feel that this needs further inspection or due diligence.

hashicorp / consul