When running filebeat as a headless service, all Consul pods break leading to all systems in the service mesh failing.

[codex70]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

Filebeat runs without a service so in order to add filebeat to the service mesh, it is necessary to create a headless service. As soon as this headless service is added to the service mesh, all Consul pods stop working and consequently any service mesh services also fail leaving the entire environment broken.

The only way I have found to fix this is to completely destroy the kubernetes clusters and start again from scratch. Obviously this is highly undesirable.

Reproduction Steps

1. Install Consul on a kubernetes cluster.
2. Install ECK Operator using the helm chart (https://github.com/elastic/cloud-on-k8s/tree/main/deploy/eck-operator), with the following values for connect-inject:

   podAnnotations: {
   consul.hashicorp.com/connect-inject: "true",
   consul.hashicorp.com/connect-service: "elastic-operator"
   }

3. Install Filebeat using the following filebeat.yaml file:

   
   apiVersion: beat.k8s.elastic.co/v1beta1
   kind: Beat
   metadata:
   name: filebeat
   spec:
   type: filebeat
   version: 8.2.3
   config:
   output.elasticsearch:
     hosts: https://127.0.0.1:8443
     username: "elastic"
     password: "password"
   
   filebeat.inputs:
     - type: container
       paths:
         - /var/log/containers/*.log
       multiline.pattern: '^{'
       multiline.negate: true
       multiline.match: after
   
   processors:
     - add_kubernetes_metadata:
         in_cluster: true
   daemonSet:
   podTemplate:
     metadata:
       labels:
         app: filebeat
       annotations:
         consul.hashicorp.com/connect-service: "filebeat"
         consul.hashicorp.com/kubernetes-service: "filebeat"
         consul.hashicorp.com/connect-inject: "true"
         consul.hashicorp.com/connect-service-upstreams: "elastic-search:9200"
   
     spec:
       automountServiceAccountToken: true
       serviceAccount: filebeat
       dnsPolicy: ClusterFirstWithHostNet
       hostNetwork: true
       securityContext:
         runAsUser: 0
       containers:
         - name: filebeat
           volumeMounts:
             - name: varlogcontainers
               mountPath: /var/log/containers
             - name: varlogpods
               mountPath: /var/log/pods
             - name: varlibdockercontainers
               mountPath: /var/lib/docker/containers
       volumes:
         - name: varlogcontainers
           hostPath:
             path: /var/log/containers
         - name: varlogpods
           hostPath:
             path: /var/log/pods
         - name: varlibdockercontainers
           hostPath:
             path: /var/lib/docker/containers

4. Consul now destroys itself and the cluster.
### Logs

<!---

Provide log files from Consul Kubernetes components by providing output from `kubectl logs` from the pod and container that is surfacing the issue. 

<details>
  <summary>Logs</summary>

output from 'kubectl logs' in relevant components

</details>
--->
I was unable to capture an relevant log details, however I was able to replicate the issue several times.

### Expected behavior

<!--- What was the expected result after following the reproduction steps? --->
Filebeat install as expected inside the service mesh and does not destroy the kubernetes cluster.
### Environment details

<!---

If not already included, please provide the following:
- `consul-k8s` version:
- `values.yaml` used to deploy the helm chart:

Additionally, please provide details regarding the Kubernetes Infrastructure, as shown below:
- Kubernetes version: v1.22.x
- Cloud Provider (If self-hosted, the Kubernetes provider utilized): EKS, AKS, GKE, OpenShift (and version), Rancher (and version), TKGI (and version)
- Networking CNI plugin in use: Calico, Cilium, NSX-T 

Any other information you can provide about the environment/deployment.
--->
Kubernetes version: v1.23.4 (although this has been replicated on earlier versions)
Consul installed using helm chart 0.48.0, although again this behaviour was seen on earlier versions.
Cloud provider OKE.

### Additional Context

<!---
Additional context on the problem. Docs, links to blogs, or other material that lead you to discover this issue or were helpful in troubleshooting the issue. 
--->

[kschoche]

Hi @codex70 - Thanks for filing this issue and i'm sorry that you're having trouble deploying. Would you be able to provide a bit more information about your configuration such as the exact values.yaml files you're using for both Consul and ECK? I was able to roughly follow your steps and it appears that everything comes online without Consul crashing. Although I suspect the lack of a Kubernetes Service in the Beat deployment will cause it to fail to connect to the mesh.

Here's what I did:

• helm install consul hashicorp/consul -f values.yaml

  
  demo $ cat val.yaml
  global:
  datacenter: "dc1"
  name: consul
  tls:
  enabled: true
  enableAutoEncrypt: true
  acls:
  manageSystemACLs: true
  server:
  replicas: 1
  connectInject:
  replicas: 1
  enabled: true
  controller:
  enabled: true
  ui:
  enabled: true
  service:
  enabled: true
  type: LoadBalancer

* `helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace -f eck.yaml`
```eck.yaml
demo $ cat eck.yaml
podAnnotations: {
    consul.hashicorp.com/connect-inject: "true",
    consul.hashicorp.com/connect-service: "elastic-operator"
}

demo $ k apply -f filebeat.yaml
beat.beat.k8s.elastic.co/filebeat unchanged
demo $ k get beat
NAME       HEALTH   AVAILABLE   EXPECTED   TYPE       VERSION   AGE
filebeat   red                             filebeat             7m19s

I suspect there are some other issues to resolve before you'll be able to make this configuration work, namely that you'll need a service (headless if necessary) that fronts your filebeat service named filebeat as well as a serviceaccount created:

[from the daemonset]
Events:
  Type     Reason        Age                 From                  Message
  ----     ------        ----                ----                  -------
  Warning  FailedCreate  81s (x18 over 12m)  daemonset-controller  Error creating: pods "filebeat-beat-filebeat-" is forbidden: error looking up service account default/filebeat: serviceaccount "filebeat" not found

[codex70]

Hi @kschoche,

Thanks for getting back to me so quickly.

The configuration for Elastic Operator is just the podAnnotations as you have above, the rest is default configuration. Elasticsearch itself is deployed in a separate cluster and the plan is to use Filebeat to connect over the service mesh.

There is a service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: Filebeat

And the headless service:

apiVersion: v1
kind: Service
metadata:
  name: filebeat
  labels:
    app: filebeat
spec:
  selector:
    app: filebeat
  clusterIP: None

Consul is slightly more complicated as we're running a federated cluster, as previously mentioned the plan is to use Filebeat to send logs to the primary cluster:

consul:
  global:
    name: consul
    datacenter: dc2
    metrics:
      enabled: true
    tls:
      enabled: true
      caCert:
        secretName: consul-federation
        secretKey: caCert
      caKey:
        secretName: consul-federation
        secretKey: caKey
    federation:
      enabled: true
      primaryDatacenter: dc1
    acls:
      manageSystemACLs: true
      replicationToken:
        secretName: consul-federation
        secretKey: replicationToken
    gossipEncryption:
      secretName: consul-gossip-encryption-key
      secretKey: key
  connectInject:
    enabled: true
    default: false
  controller:
    enabled: true
  meshGateway:
    enabled: true
  client:
    exposeGossipPorts: true 
  server:
    extraVolumes:
      - type: secret
        name: consul-federation
        items:
          - key: serverConfigJSON
            path: config.json
        load: true
    exposeGossipAndRPCPorts: true 
    ports:
      serflan:
        port: 9301
  apiGateway:
    enabled: true
    image: "hashicorp/consul-api-gateway:0.3.0"
    logLevel: debug
    managedGatewayClass:
      enabled: true
      serviceType: LoadBalancer
      useHostPorts: true
      copyAnnotations:
        service:
          annotations: |
            - external-dns.alpha.kubernetes.io/hostname
            - external-dns.alpha.kubernetes.io/ttl

  syncCatalog:
    enabled: true
    default: true
    toConsul: true
    toK8S: true
    syncClusterIPServices: false

The rest is exactly as you have it above. I would be interested to know how you get on just adding the service account and headless service. I might be able to create a separate cluster later today to try a really simple setup and see if I get the same problem.

As I say, the consul configuration is a little complicated especially as this is a secondary datacenter in a federated cluster, but I suspect this isn't the problem.

[darkn3rd]

From what I understand, a service with an advertised port is required for consul connect injector to configure envoy proxy and register the service in consul. This is required to communication to happen on the service mesh, even if it is not listening on any port.

• https://developer.hashicorp.com/consul/docs/k8s/connect#accepting-inbound-connections

[codex70]

OK, so how can a service that doesn't have an advertised port communicate in the service mesh. There are plenty of reasons why a service may not need to be open to communication, but would need to connect with a service in the mesh (any type of job like process). The only way I have been able to get it to work is to create an ingress, which seems to defeat the point.

[darkn3rd]

@codex70 This is exactly my thoughts as well, because there are such a thing as a pure client scenario. Regardless, this is a hard requirement by Consul.

The good news is that, the service does not actually need to be listening on the port, at least that is my understanding from the example in the docs. In that example, the service shows something listening on port 80, but in reality, nothing is listening as the pod runs bin/sh -c -- while true; do sleep 30; done;.

The best practices for cloud-native would be to run a small web service (like flask, sinatra, express, etc) for a health check. But alas, this is not necessary given their docs.

[codex70]

The good news is that, the service does not actually need to be listening on the port, at least that is my understanding from the example in the docs. In that example, the service shows something listening on port 80, but in reality, nothing is listening as the pod runs bin/sh -c -- while true; do sleep 30; done;.

Thats a really good idea and I will give that a try some time over the next couple of days and report back. The big issue I have is that it destroys the entire cluster if it doesn't work, so will probably have to create a new cluster where I can test things.

[codex70]

@darkn3rd I've tried again making sure the service shows as listening on port 80:

apiVersion: v1
kind: Service
metadata:
  name: filebeat-test
  labels:
    app: filebeat-test
spec:
  selector:
    app: filebeat-test
  clusterIP: None
  ports:
    - port: 80

But unfortunately it still completely destroys the entire cluster. I've updated to latest versions of consul etc. so it appears that this is still very much an issue.

[codex70]

@kschoche it would be good to know if anyone at Consul is able to replicate the problem

[darkn3rd]

I am curious what are the k8s services that are finally rendered from the Beat CRD. Unless someone at Hashicorp is familiar with this operator and CRD, it'll be harder for them to spot any issues. I am curious specifically what's rendered between a service and the headless service. For example, if they share the same ports, consul will fail. Also, check out the logs from the various consul components, and check resources as well, as consul doesn't emit to logs it is out of resources, it just spins forever in stuck state. Without playing with the Beat CRD myself, I wouldn't know where to look per say. Sounds like exciting project.

[jrhunger]

@codex70 I found this while exploring Consul versus some other service meshes and had also recently found this discussion, which made me wonder if in your federated environment part of the issue is a "filebeat" service in another cluster in the same mesh, being treated as "the same service" (whatever that would entail)?

[codex70]

@jrhunger. it was a good idea to check, but in my case, the filebeat services do have different names in each environment.

I suspect the real problem I'm having is that, by default there is no port, or endpoint that filebeat exposes. I've just checked though, and it looks like it might be possible to expose an endpoint for metrics. I will take a look and see if that would fix the issue. My big problem is that if I get it wrong, it destroys the entire Consul cluster in a totally unrecoverable way. In the end I have to completely destroy all kubernetes clusters and start again from scratch. This is obviously a real problem for Consul.

[jrhunger]

@codex70 Maybe the issue is your filebeat service is using hostNetwork: true ? Maybe then when Consul CNI sets the iptables rules to route all pod traffic through the Envoy sidecar, that it's doing it for the whole host due to lack of network isolation?

[codex70]

@jrhunger, the filebeat service is super simple:

apiVersion: v1
kind: Service
metadata:
  name: filebeat
  labels:
    app: filebeat
spec:
  selector:
    app: filebeat
  ports:
    - name: http
      port: 5506
      targetPort: 5506
      protocol: TCP

I found out that it's possible to setup filebeat with an endpoint for metrics, so used this port (5506) as the port and targetport for the service. This allows me to create a simple service which can be clusterip.

If I don't add connect inject = true for the pods annotations, this seems to work, however as soon as I try to add the filebeat service it completely destroys the whole kubernetes cluster.

It starts with all ingress controllers failing with a bad gateway 502, then eventually I get site cannot be reached.

At this point, about half of the pods on the cluster enter into a CrashLoopBackOff. It appears that anything trying to connect with Consul is unable to do so:

"error":"Get \"https://consul.service.consul:8501/v1/agent/connect/ca/roots\": dial tcp x.x.x.x:8501: connect: connection refused

[jrhunger]

@codex70 this is the part i'm talking about, in the filebeat daemonset yaml:

      spec:
        automountServiceAccountToken: true
        serviceAccount: filebeat
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true

You're running a filebeat on every node, and it is using hostNetwork, so i think injecting the consul envoy pod (and associated CNI activity) is probably causing all traffic (maybe including that from the other envoy proxies) to pass through the filebeat envoy. You could inspect the iptables rules after deployment to confirm.

[codex70]

@jrhunger well spotted, that's it. You've explained it perfectly and now I understand why it was causing problems.

It looks like I don't necessarily need:

        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true

Now I need to find out what the implications are of removing it, there may be some further setup required for Filebeat, but I now know what was happening and how to fix it!

Once again, massive thanks.