Deploying fleet / integration server should not require manually setting the basic auth realm in Kibana

[philippkahr]

Proposal

Use case. Why is this important?

When you deploy Kibana using ECK and set another realm to the order of 0, it automatically disables the built in basic realm. Take the following Kibana manifest as an example.

When you now want to deploy a fleet server, it will fail with the following error:

Requesting service_token from Kibana.
Error: erequest to get security token from Kibana failed: Unauthorized: <nil>
For help, please see our troubleshooting guide at ...

@pebrc explained:

The background for this requirement is that ECK uses the KIBANA_FLEET_USERNAME and KIBANA_FLEET_PASSWORD environment variables behind the scenes to configure Fleet Server to create a service token for itself.

From a user perspective it would be cool if ECK can do some magic to make this work when adding a fleet server for enrollment.

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: production-kibana
  namespace: elastic
spec:
  version: 8.1.2
  count: 1
  elasticsearchRef:
    name: production-elasticsearch
  config:
    saml.saml1:
       order: 0
       realm: "saml1"
     # basic.basic1:
     #   order: 1
    server.publicBaseUrl: "https://logs.myfancyes.tech"
    xpack.fleet.agents.elasticsearch.hosts: ["https://logs-es.myfancyes.com:443"]
    xpack.fleet.agents.fleet_server.hosts: ["https://logs-fleet.myfancyes.com:443"]
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: fleet_server
        version: latest
    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        is_default_fleet_server: true
        namespace: elastic
        monitoring_enabled:
          - logs
          - metrics
        package_policies:
        - name: fleet_server-1
          id: fleet_server-1
          package:
            name: fleet_server
      - name: Elastic Agent on ECK policy
        id: eck-agent
        namespace: elastic
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        is_default: true
        package_policies:
          - name: system-1
            id: system-1
            package:
              name: system

[pebrc]

We have recently reworked the way the enrolment tokens for Fleet/Elastic Agents in Fleet are generated. It is now the operator itself that is interacting with the Kibana API. The problem described in this issue remains: Fleet enrolment will fail if the basic authentication provider in Kibana is disabled.

The only "clever" thing we could do is inject the basic auth provider if the user has not specified one. This might however be undesired if they disabled it because of security requirements or company wide authentication policies etc.

We also have an implementation problem to solve in that the Kibana controller responsible for generating the kibana.yml based on the user specified values and the values ECK sets is currently not aware of Fleet or Agent. We would need to a communication channel into the agent-kb association controller to allow the association controller to express to the Kibana controller that we want to use the Kibana API and need working basic auth for it.