Add Kibana + ElasticSearch + FluentBit logs

[JMLX42]

• [x] Working Ansible role to setup an EFK (ElasticSearch + Fluent-bit + Kibana) stack.
• [x] Persistent data for ElasticSearch and Kibana (cf https://github.com/democratech/2022/issues/20#issuecomment-813676002).
• [x] Create a dedicated democratech/infra team list of GitHub users (cf https://github.com/democratech/2022/issues/20#issuecomment-813677591) that will be whitelisted by Vouch
• [x] Setup authentication for Kibana using Vouch
• [x] Disable auth by default on the local dev environment
• [x] Update the documentation.
• [x] Deploy
• [x] Confirm that https://logs.infra.laprimaire.org is accessible only after authenticating via GitHub
• [x] Confirm that Kibana works as expected to explore the logs

[JMLX42]

Apparently, the logging happens at the docker daemon level: https://github.com/docker/compose/issues/2657 So it's impossible to use a bridge network to communicate between fluent-bit and other apps.

One solution might be to add a fluent-bit container to each docker-compose stack and add them to a shared elasticsearch network.

[JMLX42]

Kibana requires the XPACK extension to enable the login form. And XPACK os not part of the OSS offer. So I'll use a Vouch reverse proxy instead:

https://github.com/vouch/vouch-proxy

[JMLX42]

One solution might be to add a fluent-bit container to each docker-compose stack and add them to a shared elasticsearch network.

This worked:

fluent-bit:
    image: fluent/fluent-bit:1.7.2
    ports:
      - 24224:24224
      - 24224:24224/udp

For each service we want the logs from, in the relevant docker-compose.yml files:

    logging:
      driver: "fluentd"
      options:
        fluentd-address: "localhost:24224"
        fluentd-async: "true"

[JMLX42]

The official Kibana documentation does not explain how to make the data persistent. And the Dockerfile has no VOLUME. So I assume all the data is stored inside ElasticSearch.

[JMLX42]

Whitelisting a whole GitHub team is not secure enough: it provides no traceability of who's authorized to access the infra.laprimaire.org subdomains protected by the Vouch proxy. Instead, we will use a whitelist of GitHub users.

This whitelist will be specified with the VOUCH_WHITELIST env var for now, but will be encrypted in the repo and editable via PR when https://github.com/democratech/2022/issues/18 is implemented.

It's important that this list remains secret so that the whitelisted accounts won't be easy targets for potential phishing attacks.

[JMLX42]

@thibauld please do the last 2 tasks:

• Confirm that https://logs.infra.laprimaire.org is accessible only after authenticating via GitHub
• Confirm that Kibana works as expected to explore the logs

[JMLX42]

Got confirmation from @thibauld on Telegram => closing.