Fleet integration fails when attempting to combine docker compose files.

[GrantNagel]

Hi there,

I'm using Unraid and features are limited when using custom docker images outside the community marketplace. Unraid has a plugin which allows you to run docker-compose commands and create a stack within the UI assigning docker images to containers. In order to populate the images and startup/stop through the UI I would like to launch fleet from the main yml.

When I try to absorb the fleet yml inside the primary yml, fleet is no longer recognised in kibana. I'm not sure how the pathing works with regards to extensions. Is there a way I can modify the main yml to include extensions? I have tried the following alteration without success.

version: '3.7'

services:

  # The 'tls' service runs a one-off script which initializes TLS certificates and
  # private keys for all components of the stack inside the local tls/ directory.
  #
  # This task only needs to be performed once, *before* the first stack startup.
  #
  # By default, it is excluded from the services started by 'docker compose up'
  # due to the non-default profile it belongs to. To run it, either provide the
  # '--profile=setup' CLI flag to Compose commands, or "up" the service by name
  # such as 'docker compose up tls'.
  tls:
    profiles:
      - setup
    container_name: elastic-tls
    build:
      context: tls/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    user: root  # ensures we can write to the local tls/ directory.
    init: true
    volumes:
      - ./tls/entrypoint.sh:/entrypoint.sh:ro,Z
      - ./tls/instances.yml:/usr/share/elasticsearch/tls/instances.yml:ro,Z
      - ./tls/certs:/usr/share/elasticsearch/tls/certs:z

  # The 'setup' service runs a one-off script which initializes users inside
  # Elasticsearch — such as 'logstash_internal' and 'kibana_system' — with the
  # values of the passwords defined in the '.env' file.
  #
  # This task is only performed during the *initial* startup of the stack. On all
  # subsequent runs, the service simply returns immediately, without performing
  # any modification to existing users.

  setup:
    container_name: elastic-setup
    build:
      context: setup/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    init: true
    volumes:
      - ./setup/entrypoint.sh:/entrypoint.sh:ro,Z
      - ./setup/helpers.sh:/helpers.sh:ro,Z
      - ./setup/roles:/roles:ro,Z
      - setup:/state:Z
      # (!) CA certificate. Generate using the 'tls' service.
      - ./tls/certs/ca/ca.crt:/ca.crt:ro,z
    environment:
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD:-}
      LOGSTASH_INTERNAL_PASSWORD: ${LOGSTASH_INTERNAL_PASSWORD:-}
      KIBANA_SYSTEM_PASSWORD: ${KIBANA_SYSTEM_PASSWORD:-}
      METRICBEAT_INTERNAL_PASSWORD: ${METRICBEAT_INTERNAL_PASSWORD:-}
      FILEBEAT_INTERNAL_PASSWORD: ${FILEBEAT_INTERNAL_PASSWORD:-}
      HEARTBEAT_INTERNAL_PASSWORD: ${HEARTBEAT_INTERNAL_PASSWORD:-}
      MONITORING_INTERNAL_PASSWORD: ${MONITORING_INTERNAL_PASSWORD:-}
      BEATS_SYSTEM_PASSWORD: ${BEATS_SYSTEM_PASSWORD:-}
    networks:
      - elk
    depends_on:
      - elasticsearch

  elasticsearch:
    container_name: elasticsearch
    build:
      context: elasticsearch/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    volumes:
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro,Z
      - elasticsearch:/usr/share/elasticsearch/data:Z
      # (!) TLS certificates. Generate using the 'tls' service.
      - ./tls/certs/ca/ca.crt:/usr/share/elasticsearch/config/ca.crt:ro,z
      - ./tls/certs/elasticsearch/elasticsearch.crt:/usr/share/elasticsearch/config/elasticsearch.crt:ro,z
      - ./tls/certs/elasticsearch/elasticsearch.key:/usr/share/elasticsearch/config/elasticsearch.key:ro,z
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      node.name: elasticsearch
      ES_JAVA_OPTS: -Xms512m -Xmx512m
      # Bootstrap password.
      # Used to initialize the keystore during the initial startup of
      # Elasticsearch. Ignored on subsequent runs.
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD:-}
      # Use single node discovery in order to disable production mode and avoid bootstrap checks.
      # see: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
      discovery.type: single-node
    networks:
      - elk

  logstash:
    container_name: logstash
    build:
      context: logstash/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro,Z
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro,Z
      # (!) CA certificate. Generate using the 'tls' service.
      - ./tls/certs/ca/ca.crt:/usr/share/logstash/config/ca.crt:ro,z
    ports:
      - 5044:5044
      - 50000:50000/tcp
      - 50000:50000/udp
      - 9600:9600
    environment:
      LS_JAVA_OPTS: -Xms256m -Xmx256m
      LOGSTASH_INTERNAL_PASSWORD: ${LOGSTASH_INTERNAL_PASSWORD:-}
    networks:
      - elk
    depends_on:
      - elasticsearch

  kibana:
    container_name: kibana
    build:
      context: kibana/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro,Z
      # (!) TLS certificates. Generate using the 'tls' service.
      - ./tls/certs/ca/ca.crt:/usr/share/kibana/config/ca.crt:ro,z
      - ./tls/certs/kibana/kibana.crt:/usr/share/kibana/config/kibana.crt:ro,Z
      - ./tls/certs/kibana/kibana.key:/usr/share/kibana/config/kibana.key:ro,Z
    ports:
      - 5601:5601
    environment:
      KIBANA_SYSTEM_PASSWORD: ${KIBANA_SYSTEM_PASSWORD:-}
    networks:
      - elk
    depends_on:
      - elasticsearch

  fleet-server:
    container_name: fleet
    build:
      context: extensions/fleet/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    volumes:
      - fleet-server:/usr/share/elastic-agent/state:Z
      # (!) TLS certificates. Generate using the 'tls' service.
      - ./tls/certs/ca/ca.crt:/usr/share/elastic-agent/ca.crt:ro,z
      - ./tls/certs/fleet-server/fleet-server.crt:/usr/share/elastic-agent/fleet-server.crt:ro,Z
      - ./tls/certs/fleet-server/fleet-server.key:/usr/share/elastic-agent/fleet-server.key:ro,Z
    environment:
      FLEET_SERVER_ENABLE: '1'
      FLEET_SERVER_POLICY_ID: fleet-server-policy
      FLEET_URL: https://fleet-server:8220
      FLEET_SERVER_CERT: /usr/share/elastic-agent/fleet-server.crt
      FLEET_SERVER_CERT_KEY: /usr/share/elastic-agent/fleet-server.key
      ELASTICSEARCH_HOST: https://elasticsearch:9200
      ELASTICSEARCH_CA: /usr/share/elastic-agent/ca.crt
      # Fleet plugin in Kibana
      KIBANA_FLEET_SETUP: '1'
      # Enrollment.
      # (a) Auto-enroll using basic authentication
      ELASTICSEARCH_USERNAME: elastic
      ELASTICSEARCH_PASSWORD: ${ELASTIC_PASSWORD:-}
      # (b) Enroll using a pre-generated service token
      #FLEET_SERVER_SERVICE_TOKEN: <service_token>
    ports:
      - 8220:8220
    hostname: fleet-server
    # Elastic Agent does not retry failed connections to Kibana upon the initial enrollment phase.
    restart: on-failure
    networks:

      - elk
    depends_on:
      - elasticsearch
      - kibana

networks:
  elk:
    driver: bridge

volumes:
  setup:
  elasticsearch:
  fleet-server:

[antoineco]

Could you please share the logs? At the very least the ones from Fleet.

[GrantNagel]

{"log.level":"info","@timestamp":"2022-12-15T00:38:49.638Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:38:49Z - message: Application: fleet-server--8.5.3[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:38:50.138Z","log.origin":{"file.name":"status/reporter.go","file.line":326},"message":"Elastic Agent status changed to \"error\": \"app fleet-server--8.5.3-77c54199: Error - EOF\"","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:38:50.138Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-12-15T00:38:50Z - message: Application: fleet-server--8.5.3[]: State changed to FAILED: Error - EOF - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-15T00:39:00.139Z","log.origin":{"file.name":"status/reporter.go","file.line":326},"message":"Elastic Agent status changed to \"online\": \"\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:00.139Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:00Z - message: Application: fleet-server--8.5.3[]: State changed to RESTARTING:  - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:00.139Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:00Z - message: Application: fleet-server--8.5.3[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:00.139Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:00Z - message: Application: fleet-server--8.5.3[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:39:00.150Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"fleet-server stderr: \"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-12-15T00:39:00Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-12-15T00:39:00Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n\"","agent.console.name":"fleet-server","agent.console.type":"stderr","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:00.656Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:00Z - message: Application: fleet-server--8.5.3[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:39:01.156Z","log.origin":{"file.name":"status/reporter.go","file.line":326},"message":"Elastic Agent status changed to \"error\": \"app fleet-server--8.5.3-77c54199: Error - EOF\"","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:39:01.156Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-12-15T00:39:01Z - message: Application: fleet-server--8.5.3[]: State changed to FAILED: Error - EOF - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-15T00:39:11.157Z","log.origin":{"file.name":"status/reporter.go","file.line":326},"message":"Elastic Agent status changed to \"online\": \"\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:11.157Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:11Z - message: Application: fleet-server--8.5.3[]: State changed to RESTARTING:  - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:11.157Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:11Z - message: Application: fleet-server--8.5.3[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:11.157Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:11Z - message: Application: fleet-server--8.5.3[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:39:11.167Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"fleet-server stderr: \"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-12-15T00:39:11Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-12-15T00:39:11Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n\"","agent.console.name":"fleet-server","agent.console.type":"stderr","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:11.673Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:11Z - message: Application: fleet-server--8.5.3[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:39:12.174Z","log.origin":{"file.name":"status/reporter.go","file.line":326},"message":"Elastic Agent status changed to \"error\": \"app fleet-server--8.5.3-77c54199: Error - EOF\"","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:39:12.174Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-12-15T00:39:12Z - message: Application: fleet-server--8.5.3[]: State changed to FAILED: Error - EOF - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-15T00:39:22.175Z","log.origin":{"file.name":"status/reporter.go","file.line":326},"message":"Elastic Agent status changed to \"online\": \"\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:22.175Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:22Z - message: Application: fleet-server--8.5.3[]: State changed to RESTARTING:  - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:22.175Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:22Z - message: Application: fleet-server--8.5.3[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:22.175Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:22Z - message: Application: fleet-server--8.5.3[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:39:22.186Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"fleet-server stderr: \"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-12-15T00:39:22Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-12-15T00:39:22Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n\"","agent.console.name":"fleet-server","agent.console.type":"stderr","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:22.691Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:22Z - message: Application: fleet-server--8.5.3[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:39:23.191Z","log.origin":{"file.name":"status/reporter.go","file.line":326},"message":"Elastic Agent status changed to \"error\": \"app fleet-server--8.5.3-77c54199: Error - EOF\"","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:39:23.191Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-12-15T00:39:23Z - message: Application: fleet-server--8.5.3[]: State changed to FAILED: Error - EOF - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-15T00:39:33.192Z","log.origin":{"file.name":"status/reporter.go","file.line":326},"message":"Elastic Agent status changed to \"online\": \"\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:33.192Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:33Z - message: Application: fleet-server--8.5.3[]: State changed to RESTARTING:  - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:33.193Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:33Z - message: Application: fleet-server--8.5.3[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:33.193Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:33Z - message: Application: fleet-server--8.5.3[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:39:33.201Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"fleet-server stderr: \"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-12-15T00:39:33Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-12-15T00:39:33Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n\"","agent.console.name":"fleet-server","agent.console.type":"stderr","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-15T00:39:33.708Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-12-15T00:39:33Z - message: Application: fleet-server--8.5.3[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:39:34.208Z","log.origin":{"file.name":"status/reporter.go","file.line":326},"message":"Elastic Agent status changed to \"error\": \"app fleet-server--8.5.3-77c54199: Error - EOF\"","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-12-15T00:39:34.208Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-12-15T00:39:34Z - message: Application: fleet-server--8.5.3[]: State changed to FAILED: Error - EOF - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}

[antoineco]

@GrantNagel I copied and pasted your exact Compose file, and performed the following steps:

1. Ran docker-compose up tls
2. Copied the CA fingerprint to kibana/config/kibana.yml

   diff --git a/kibana/config/kibana.yml b/kibana/config/kibana.yml
   index f7aaa20..6014bc9 100644
   --- a/kibana/config/kibana.yml
   +++ b/kibana/config/kibana.yml
   @@ -42,7 +42,7 @@ xpack.fleet.outputs:
       type: elasticsearch
       hosts: [ https://elasticsearch:9200 ]
       # Set to output of 'docker-compose up tls'. Example:
   -    #ca_trusted_fingerprint: 846637d1bb82209640d31b79869a370c8e47c2dc15c7eafd4f3d615e51e3d503
   +    ca_trusted_fingerprint: 92e5b27b2a30c261ebf468fdbbda5c5fb78d17504bd1f539dc002fc88a9841bb
       is_default: true
       is_default_monitoring: true

3. Ran docker-compose up

Fleet started and registered without any issue:

The error message Error - EOF is unfortunately too cryptic for me to be able to suggest any remediation step here, besides docker-compose down -v then try again.

[GrantNagel]

@antoineco Thanks for the feedback. Unfortunately, this does not work with my setup, however, I have found a workaround which involves running a script preventing the need to cd to the directory and run the stack each time. The only problem is I can't set an icon for my containers.

#!/bin/bash
(cd /mnt/path/to/docker-elk;docker compose -f docker-compose.yml -f extensions/fleet/fleet-compose.yml up)