ECK Support For Elastic Fleet With HTTP TLS Disabled Mode For ISTIO Service Mesh

[deiberts86]

Proposal

• Propose to have ECK support Fleet with "HTTP TLS Mode disabled" for ISTIO service mesh.

Use case. Why is this important?

• ISTIO service mesh provides great layer of security with ENVOY proxy sidecars. Docs for ISTIO with Elasticsearch and Kibana works perfectly within the mesh. However, Fleet requires HTTPS for Kubernetes deployments with ECK. SNI passthrough doesn't seem to work because the Elasticsearch and Kibana are running with HTTP TLS disabled flags to allow sidecars to handle the TLS traffic. The Elasticsearch service running in HTTP mode within the mesh doesn't seem to accept traffic from Fleet pod since it's running TLS at the pod level.

• We do TLS termination for the Elasticsearch service and Kibana at the edge of the mesh with the ISTIO gateway ingress. From there, sidecars are handling the mTLS traffic between containers / services. We also tried Fleet as a standalone server in front of the Kubernetes cluster with similar results. The Elastic agents managed by fleet show "Healthy" but can't receive logs. This was the case with Fleet inside the mesh or outside on the standalone server.

• I think this will be a great addition to support this within ISTIO as government entities are requiring a service mesh for "ZeroTrust" design.

Tried this link with certain Kibana options and didn't work: https://github.com/elastic/kibana/issues/47482

Discussion was done on the Elastic Slack Channel for ECK: https://app.slack.com/client/TNLBGCXTQ/CS9KR083S/thread/CS9KR083S-1662395565.799459

[naemono]

@deiberts86 I am going to try and find an option that may work here, but having more information about your environment would be helpful.

• kubernetes version
• environment (aks, eks, gke)?
• istio version
• using istio-injection for sidecar injection I assume?
• Elasticsearch version

[deiberts86]

@naemono here is what I have currently in production:

• kubernetes version = v1.22.9+rke2r2 with SUSE Rancher Federal's flavor of k3s (RKE2)
• environment (aks, eks, gke)? = BareMetal (VMware) on Prem
• istio version = 1.13.3
• using istio-injection for sidecar injection I assume? = Yes, The label is applied at the namespace layer. All Elastic pods have the sidecars injected except for daemonsets (agents or beats).
• Elasticsearch version = 8.4.0

[naemono]

@deiberts86 I seem to have finally gotten this to work with ECK handling the TLS setup for Fleet Server with the below manifest, which adds annotations to disable istio from handling mtls on the fleet server agent:

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-agent
  labels:
    app: fleet
    version: 0.0.2
spec:
  version: 8.4.0
  mode: fleet
  fleetServerEnabled: true
  deployment:
    podTemplate:
      metadata:
        annotations:
          traffic.sidecar.istio.io/includeInboundPorts: "*"
          traffic.sidecar.istio.io/excludeOutboundPorts: "8220" 
          traffic.sidecar.istio.io/excludeInboundPorts: "8220"
        labels:
          app: fleet
          version: 0.0.2
      spec:
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        serviceAccountName: fleet-server
    replicas: 1
  elasticsearchRefs:
  - name: testing
  # The below does not work for disabling tls on fleet server
  # http:
  #   tls:
  #     selfSignedCertificate:
  #       disabled: true
  kibanaRef:
    name: testing

This allows the agents to successfully check into fleet server, and be managed in the Kibana UI.

I understand that ECK not managing fleet's TLS so that Istio's automatic MTLS can manage it is a better option, similar to how Elasticsearch, and Kibana can have tls disabled. I'll gather some information about whether allowing for disabling Fleet's ECK-managed TLS would be something we would consider, and update this issue.

[deiberts86]

@naemono thank you sir for rolling through this! This is exactly what I was requesting. I'll wait for the feature to be implemented in the upcoming releases to test this out. With TLS termination done at the edge, would we have the Kibana "fleet" URL pointed to the ISTIO-Gateway? I assume yes if TLS is turned off and having sidecars handle the TLS traffic.