Fleet agent never associates with kibana if Nodeport is provided in kibana manifest

[sagargulabani]

If we set the service and nodeport configuration for kibana, the fleet server 'Agent' custom resource never associates with kibana

    # service:
    #   spec:
    #     type: NodePort
    #     ports:
    #       - port: 5601
    #         nodePort: 32000

Fleet server manifest

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server
  namespace: elastic-system
spec:
  version: 8.13.4
  kibanaRef:
    name: log-kibana
  elasticsearchRefs:
  - name: log-cluster
  mode: fleet
  fleetServerEnabled: true
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        containers:
          - name: agent
            resources:
              requests:
                memory: 1Gi
                cpu: 400m
              limits:
                memory: 1Gi
                cpu: 400m

Kibana manifest

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: log-kibana
  namespace: elastic-system
spec:
  version: 8.13.4
  count: 1
  elasticsearchRef:
    name: log-cluster
  podTemplate:
    metadata:
      namespace: elastic-system
    spec:
      containers:
        - name: kibana
          resources:
            requests:
              memory: 3Gi
              cpu: 1.5
            limits:
              memory: 4Gi
              cpu: 2
          readinessProbe:
            failureThreshold: 3
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            exec:
              command:
                - /bin/bash
                - -c
                - curl  http://127.0.0.1:5601/login -k -s
  http:
    tls:
      selfSignedCertificate:
        disabled: true
    service:
      spec:
        type: NodePort
        ports:
          - port: 5601
            nodePort: 32000
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["http://log-cluster-es-http.elastic-system.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.elastic-system.svc:8220"]
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: fleet_server
        version: latest
      - name: apm
        version: latest
      - name: log
        version: latest
      - name: kubernetes
        version: latest
    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        is_default_fleet_server: true
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        package_policies:
        - name: fleet_server-1
          id: fleet_server-1
          package:
            name: fleet_server
      - name: Elastic Agent policy Three
        id: eck-agent-three
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        is_default: true
        package_policies:
          - name: system-1
            id: system-1
            package:
              name: system
          - name: kubernetes-2
            id: kubernetes-2
            package:
              name: kubernetes
            inputs:
            - type: filestream
              enabled: true
              streams:
              - enabled: true
                data_stream:
                  type: logs
                  dataset: kubernetes.container_logs

Events:
  Type     Reason                   Age                From                                 Message
  ----     ------                   ----               ----                                 -------
  Warning  AssociationError         20s (x5 over 20s)  agent-controller                     Association backend for elasticsearch is not configured
  Warning  AssociationError         20s (x8 over 20s)  agent-controller                     Association backend for kibana is not configured
  Normal   AssociationStatusChange  20s                agent-kibana-association-controller  Association status changed from [] to [Pending]
  Normal   AssociationStatusChange  20s                agent-es-association-controller      Association status changed from [] to [elastic-system/log-cluster: Established]

It gets stuck in this state.

[pyaillet]

I had the same problem.

I was able to solve it by looking at the logs. They were showing an error related to the unavailability of a port named https on the service.

So the following modification should do the trick (at least, it did for me):

    # service:
    #   spec:
    #     type: NodePort
    #     ports:
    #       - name: https
    #         port: 5601
    #         nodePort: 32000