Support externalRef in Elastic Agent

[pebrc]

Users wantig to use elasticsearchRef, kibanaRef and or fleetServerRef to connect to either Kibana and Elasticsearch or Fleet Server runing outside the k8s cluster wil encounter a range of confusing errors and will not be able to make it work. This issue intends to document the current limitation in case we want to lift them.

1. the client we use to create enrollment tokens does not trust certificates issues by one of the well known Root CAs (to be fixed in original PR #5846)
2. the check for Kibana availability in fleet.go interacts with the K8s API to find out about Kibana's status, that is not possible with an external ref
3. external Refs do not support service account tokens yet, however Elastic Agent will try to contact Kibana for a service account token in the absence of it.

However there is already a superior alternative to using the *Ref mechanism by using enrollment tokens directly through the Pods/Elastic Agents environment:

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata: 
  name: elastic-agent
spec:
  version: 8.3.1
  mode: fleet
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        hostNetwork: true
        dnsPolicy: ClusterFirstWithHostNet
        automountServiceAccountToken: true
        securityContext:
         runAsUser: 0
        containers:
        - name: agent
          env:
          - name: FLEET_ENROLL
            value: "1"
          - name: FLEET_URL
            value: https://my-external.fleet.server
          - name: FLEET_ENROLLMENT_TOKEN
            value: --redacted--

[abdul90082]

Hi. I've been trying to setup an agent as you described here without *Ref but I am getting this error:

{"log.level":"error","@timestamp":"2023-01-11T14:33:14.742Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":204},"message":"failed to dispatch actions, error: fail to communicate with Fleet Server API client hosts: all hosts failed: 1 error occurred:\n\t* requester 0/1 to host https://fleet-server-http.elasticsearch.svc:8220/ errored: Get \"https://fleet-server-http.elasticsearch.svc:8220/api/status?\": lookup fleet-server-http.elasticsearch.svc on 10.43.0.10:53: no such host\n\n","ecs.version":"1.6.0"}

My agent config:

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent
  namespace: elasticsearch-agent
spec:
  daemonSet:
    podTemplate:
      spec:
        automountServiceAccountToken: true
        containers:
        - env:
          - name: FLEET_ENROLL
            value: "1"
          - name: FLEET_URL
            value: https://fleet-server.example.com
          - name: FLEET_ENROLLMENT_TOKEN
            value: xxxx
          name: agent
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        securityContext:
          runAsUser: 0
        serviceAccountName: elastic-agent
  mode: fleet
  version: 8.5.3

It seems it still tries to connect to a fleet server internally. Any idea how to disable this? Thanks