Implement `conf.toml` Helm generation on-the-fly based on the domain (#954)

SleepySquash commented 1 week ago

Resolves #954

Synopsis

Helm should allow specifying different configuration for different domains.

Solution

This PR adds:

nginx returning conf.toml based on the $host;
Helm generating such .yaml files based on the provided values.

1) The $host based retrieval of conf.toml is the easy part

  location = /conf {
    root       /var/www;
    expires    1m;
    try_files  /conf/$host.yaml /conf/$host.toml $uri.toml $uri.yaml $uri  =404;
  }

-> nginx will try to get the $host.toml file first, and if fails to do so, then fallbacks to conf.toml. E.g. if we are localhost, then nginx will try to return localhost.toml instead of conf.toml by default.

2) The Helm part is a bit harder. ~We need to cycle through ingress.hosts specified in the values and form a .toml file for each host being a merge of the common .Values.conf and a patch configuration for the provided host. I've managed to accomplish that by using initContainers for every host. The containers use the mergeOverwrite Helm function to merge values. And then using the simple toToml I echo the merged values into the emptyDir mount I then mount into the app container.~ See https://github.com/team113/messenger/pull/1041#discussion_r1644711821 for details.

So, given the following values.yaml file:

ingress:
  hosts:
    - localhost
    - example.com
    - empty

conf:
  server:
    http:
      url: https://domain.com
      port: 443
    ws:
      url: wss://domain.com
      port: 443
  localhost:
    legal:
      copyright: sup
  example.com:
    link:
      prefix: https://example.com/prefix
    server:
      http:
        port: 49000
      ws:
        port: 49001

(notice the empty domain having no patch, localhost domain adding the legal part and the example.com overwriting the ports)

We get the following files: localhost.yaml

server:
  http:
    port: 443
    url: https://domain.com
  ws:
    port: 443
    url: wss://domain.com
legal:
  copyright: sup

example.com.yaml

server:
  http:
    port: 49000
    url: https://domain.com
  ws:
    port: 49001
    url: wss://domain.com
link:
  prefix: https://example.com/prefix

empty.yaml

server:
  http:
    port: 443
    url: https://domain.com
  ws:
    port: 443
    url: wss://domain.com

Checklist

Created PR:
- [x] In draft mode
- [x] Name contains issue reference
- [x] Has type and k:: labels applied
Before review:
- [x] Documentation is updated (if required)
- [x] Tests are updated (if required)
- [x] Changes conform code style
- [x] CHANGELOG entry is added (if required)
- [x] FCM (final commit message) is posted or updated
- [x] Draft mode is removed
[x] Review is completed and changes are approved
- [x] FCM (final commit message) is approved
Before merge:
- [x] Milestone is set
- [x] PR's name and description are correct and up-to-date
- [x] All temporary labels are removed

SleepySquash commented 1 week ago

FCM

Allow tuning remote configuration per individual host in Helm chart (#1041, #954)

- add `conf-renderer` container and `template.sh` script generating YAML configuration per Ingress host
- support YAML format in `Config.init()` for remote configuration parsing
- use `try_files` for `/conf` Nginx route to return `$host` based configuration

SleepySquash commented 1 week ago

Discussed: current solution creates separate init container for each domain, which is NOT the desired behaviour, as this may eventually break the Helm chart deployment with more and more domains being added (imagine 1000 init containers). Ideally we should have a single init container accepting the Values context and then separating the different configurations for each domain, so that Helm doesn't embed a lot of stuff in the deployment resource.

team113 / messenger