Setting kafka config through env vars doesn't work

[invernizzie]

Repro steps:

Run: KAPACITOR_KAFKA_0_ID="kafka-cluster" KAPACITOR_KAFKA_0_BROKERS_0="192.168.99.100:31092" KAPACITOR_KAFKA_0_ENABLED=true kapacitord

Kapacitor logs this and exits:

run: create server: invalid configuration: kafka: no brokers specified, must provide at least one broker URL. To generate a valid configuration file run `kapacitord config > kapacitor.generated.conf`.

This is useful for easily configuring kapacitor running as a docker container.

[jzakrzeski]

Not sure if this helps, but I use this functionality to configure kafka for kapacitor in a docker container and it works fine, but setting those actually exports them as env vars.

The config overrides expect that the variables are in the environment, what you're doing here doesn't actually set them as environment variables, so it's likely they aren't available to a subprocess that needs them to work. The overrides are applied from a call to os.getEnv() so if you export these first you should have better luck:

export KAPACITOR_KAFKA_0_ID="kafka-cluster"; export KAPACITOR_KAFKA_0_BROKERS_0="192.168.99.100:31092"; export KAPACITOR_KAFKA_0_ENABLED=true; kapacitord

Using compose:

kapacitor:
  image: kapacitor:1.5
  environment:
    - KAPACITOR_KAFKA_0_ID="kafka-cluster"
    - KAPACITOR_KAFKA_0_BROKERS_0="192.168.99.100:31092"
    - KAPACITOR_KAFKA_0_ENABLED=true

Or docker run:

docker run -e KAPACITOR_KAFKA_0_ID="kafka-cluster" -e KAPACITOR_KAFKA_0_BROKERS_0="192.168.99.100:31092" -e KAPACITOR_KAFKA_0_ENABLED=true kapacitor:1.5

[timhallinflux]

You may also need to add an empty [[kafka]] config section to your configuration file. Then it should work.

[danatinflux]

If there's a way to run CURL commands against the kapacitor instance in question, there's a way to do it with the API:

First example: adding a broker and enabling the 'default' kafka config:

curl -v -XPOST "http://localhost:9092/kapacitor/v1/config/kafka/" -d '{"set":{"brokers":["test.test.org:9999"],"enabled":true}}'

Next, add a brand new kafka queue to kapacitor:

curl -v -XPOST "http://localhost:9092/kapacitor/v1/config/kafka/" -d '{"add":{"brokers":["test2.test.org:9999"],"id":"octopus","enabled":true}}'

This will give you this:

[root@kapa8 kapacitor]# curl -q -XGET "http://localhost:9092/kapacitor/v1/config/kafka" | jq
{
  "link": {
    "rel": "self",
    "href": "/kapacitor/v1/config/kafka"
  },
  "elements": [
    {
      "link": {
        "rel": "self",
        "href": "/kapacitor/v1/config/kafka/default"
      },
      "options": {
        "batch-size": 0,
        "batch-timeout": "0s",
        "brokers": [
          "test.test.org:9999"
        ],
        "enabled": true,
        "id": "default",
        "insecure-skip-verify": false,
        "ssl-ca": "",
        "ssl-cert": "",
        "ssl-key": "",
        "timeout": "0s",
        "use-ssl": false
      },
      "redacted": null
    },
    {
      "link": {
        "rel": "self",
        "href": "/kapacitor/v1/config/kafka/octopus"
      },
      "options": {
        "batch-size": 0,
        "batch-timeout": "0s",
        "brokers": [
          "test2.test.org:9999"
        ],
        "enabled": true,
        "id": "octopus",
        "insecure-skip-verify": false,
        "ssl-ca": "",
        "ssl-cert": "",
        "ssl-key": "",
        "timeout": "0s",
        "use-ssl": false
      },
      "redacted": null
    }
  ]
}