search

elastic / cloud-on-k8s

Elastic Cloud on Kubernetes
Other
58 stars 708 forks source link

Unable to set INSECURE mode on fleet server or set CA certs via chart #7383

Closed sevaho closed 2 months ago

sevaho commented 11 months ago

Proposal

Use case. Why is this important?

This is important as the 'out of the box' experience is not great to install ECK on Kubernetes in my opinion. While the helm chart is clean and structured, it lacks key aspects to create a working ECK stack - fleet server and agent will produce x509: certificate signed by unknown authority errors if you use the values.yaml found here: https://github.com/elastic/cloud-on-k8s/blob/main/deploy/eck-stack/examples/agent/fleet-agents.yaml and add agents via the ui.

Bug Report

What did you do?

Installed the helm chart with following values.yaml:

eck-elasticsearch:
  enabled: true
  fullnameOverride: elasticsearch
  annotations:
    eck.k8s.elastic.co/license: basic
  volumeClaimDeletePolicy: DeleteOnScaledownOnly
  nodeSets:
    - name: master
      count: 1
      podTemplate:
        spec:
          initContainers:
            - name: sysctl
              securityContext:
                privileged: true
                runAsUser: 0
              command:
                - sh
                - "-c"
                - sysctl -w vm.max_map_count=262144
          volumes:
            - name: elasticsearch-data
              emptyDir: {}
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 500Gi
eck-kibana:
  enabled: true
  fullnameOverride: kibana
  annotations:
    eck.k8s.elastic.co/license: basic
    teleport.dev/protocol: https
  spec:
    elasticsearchRef:
      name: elasticsearch
    config:
      server.publicBaseUrl: https://logs.asic.li
      xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.elastic-system.svc:9200"]

      xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.elastic-system.svc:8220"]
      xpack.fleet.packages:
        - name: system
          version: latest
        - name: elastic_agent
          version: latest
        - name: fleet_server
          version: latest
        - name: kubernetes
          version: latest
      xpack.fleet.agentPolicies:
        - name: Fleet Server on ECK policy
          id: eck-fleet-server
          namespace: default
          monitoring_enabled:
            - logs
            - metrics
          package_policies:
            - name: fleet_server-1
              id: fleet_server-1
              package:
                name: fleet_server
        - name: Elastic Agent on ECK policy
          id: eck-agent
          namespace: default
          monitoring_enabled:
            - logs
            - metrics
          unenroll_timeout: 900
          package_policies:
            - package:
                name: system
              name: system-1
            - package:
                name: kubernetes
              name: kubernetes-1
eck-fleet-server:
  enabled: true
  annotations:
    eck.k8s.elastic.co/license: basic
  fullnameOverride: "fleet-server"
  spec:
    # Agent policy to be used.
    policyID: eck-fleet-server
    kibanaRef:
      name: kibana
    elasticsearchRefs:
      - name: elasticsearch

What did you expect to see?

I am looking at the following values.yaml file to see documentation on what can be configured: https://github.com/elastic/cloud-on-k8s/blob/main/deploy/eck-stack/charts/eck-fleet-server/values.yaml

  1. I would like to see a way to set ssl CA certs via the chart, the elastic documentation says you need to set it via ssl.certificate_authorities, but I don't see any option. see: https://www.elastic.co/guide/en/fleet/8.11/secure-connections.html#generate-fleet-server-certs

Unable to set env vars in chart, no documentation or directive to set env vars, I wanted to set: FLEET_INSECURE and FLEET_SERVER_INSECURE_HTTP to true as found here: https://github.com/elastic/elastic-agent/blob/main/internal/pkg/agent/cmd/setup_config.go#L79 . But I don't have the option to provide it in the chart.

What did you see instead? Under which circumstances?

Environment

Kubernetes

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.26.10+rke2r2

hiroki783 commented 2 months ago

Based on this document, I was able to override the environment variables by modifying the values.yaml file.

after

root@v0-dev-01:~/project/elasticsearch/cloud-on-k8s# kubectl exec -it eck-stack-with-fleet-eck-agent-agent-84thl -n elastic-stack -- /bin/bash root@v0-k8s-01:/usr/share/elastic-agent# export | grep -E "FLEET_CA" declare -x FLEET_CA=""



If you are looking to register elastic agents both inside and outside of ECK, [my blog](https://hirohirolab.com/en/blog/2024/08/series_elasticsearch_how_to_register_agent_outside_of_eck/) or https://github.com/elastic/elastic-agent/issues/2762 might be helpful.
rhr323 commented 2 months ago

Hey @sevaho ,

As pointed out by @hiroki783, FLEET_INSECURE, FLEET_SERVER_INSECURE_HTTP (and any other environment variable) can be configured via the podTemplate section via adding environment variables.

    podTemplate:
      spec:
        securityContext:
          runAsUser: 0 
            containers:
            - name: agent
              env:
              - name: FLEET_INSECURE
                value: true

@hiroki783: thanks for linking your blog post, nice work!