Fleet server pod not created when Kibana is set with `SERVER_BASEPATH`

[msapitree]

Bug Report

What did you do?

Following quickstart 1) Deployed ECK 2) Deployed Elastic resource 3) Deployed Kibana resource 4) Redeployed Kibana resource with config modification (to setup public access to Kibana via k8s ingress on https://xxx.mydomain.com/monitoring/kibana): 5) Deployed Fleet Server Agent (together with ClusterRole, ClusterRoleBinding and ServiceAccount)

What did you expect to see?

• pod is created for the fleet-server agent
• no error in operator logs
• agent is reported as green

What did you see instead? Under which circumstances?

• error in operator log failed to request http://monitoring-kb-http.monitoring-at.svc:5601/api/fleet/setup, status is 404)

  Due to the change to server context, the URL path /api/fleet/setup is no longer valid and should have become /monitoring/kibana/api/fleet/setup considering the server.basePath config parameter (or it's env variable equivalent).

• no health information on elastic resource

  
  $kubectl -n monitoring-at get elastic
  NAME                                      HEALTH   AVAILABLE   EXPECTED   VERSION   AGE
  agent.agent.k8s.elastic.co/fleet-server                                             109m

NAME HEALTH NODES VERSION AGE apmserver.apm.k8s.elastic.co/monitoring green 1 8.14.0 5d21h

NAME HEALTH NODES VERSION PHASE AGE elasticsearch.elasticsearch.k8s.elastic.co/monitoring green 3 8.14.0 Ready 5d22h

NAME HEALTH NODES VERSION AGE kibana.kibana.k8s.elastic.co/monitoring green 1 8.14.0 114m

**Environment**

* ECK version: 2.13.0

* Kubernetes information: AKS

Client Version: v1.29.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.9

* Resource definition:
- Kibana with `SERVER_BASEPATH`

```yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: monitoring
  namespace: monitoring-at
spec:
  version: 8.14.0
  count: 1
  elasticsearchRef:
    name: monitoring
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  podTemplate:
    spec:
      containers:
      - env:
        - name: SERVER_BASEPATH
          value: /monitoring/kibana
        - name: SERVER_REWRITEBASEPATH
          value: "true"
        name: kibana

• Fleet Server

  apiVersion: agent.k8s.elastic.co/v1alpha1
  kind: Agent
  metadata:
  name: fleet-server
  spec:
  version: 8.14.0
  kibanaRef:
  name: monitoring
  elasticsearchRefs:
  - name: monitoring
  mode: fleet
  fleetServerEnabled: true
  policyID: eck-fleet-server
  deployment:
  replicas: 1
  podTemplate:
    spec:
      securityContext:
        runAsUser: 0
      serviceAccountName: fleet-server
      automountServiceAccountToken: true

• Logs:

  {
  "log.level": "error",
  "@timestamp": "2024-06-18T20:35:39.770Z",
  "log.logger": "manager.eck-operator",
  "message": "Reconciler error",
  "service.version": "2.13.0+8896afe1",
  "service.type": "eck",
  "ecs.version": "1.4.0",
  "controller": "agent-controller",
  "object": {
  "name": "fleet-server",
  "namespace": "monitoring-at"
  },
  "namespace": "monitoring-at",
  "name": "fleet-server",
  "reconcileID": "958131c8-738f-4062-b63e-0fcc831f297e",
  "error": "failed to request http://monitoring-kb-http.monitoring-at.svc:5601/api/fleet/setup, status is 404)",
  "errorCauses": [
  {
    "error": "failed to request http://monitoring-kb-http.monitoring-at.svc:5601/api/fleet/setup, status is 404)"
  }
  ],
  "error.stack_trace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227"
  }

[pebrc]

What you are trying to do is currently not supported. The operator, as you have found out is not aware of the server.basePath setting. The only work around I can think of is to try to do the rewrite proxy side with an ingress implementation specific path (e.g for nginx)

We could potentially inspect the Kibana config and adjust the paths. But it is a bit tricky your example shows. You are using environment variables while the more idiomatic way would be to use the config attribute:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 8.14.0
  count: 1
  elasticsearchRef:
    name: elasticsearch
  config:
    server:
      basePath: /kibana
      rewriteBasePath: true

A potential "fix" for this bug would probably only look at the config element.