search

influxdata / kapacitor

Open source framework for processing, monitoring, and alerting on time series data
MIT License
2.31k stars 493 forks source link

ServerID is not accounted when consuming subscription #2038

Open stanislav-zaprudskiy opened 6 years ago

stanislav-zaprudskiy commented 6 years ago

We run Kapacitor as a Docker container (FROM kapacitor:1.5.1). And as we see no need to persist any "runtime" data, we don't use any dedicated volume for it.

However, what we sometimes do is change Kapacitor configuration (conf, handlers, alert definitions), which is then followed by image re-build and container re-run, while Kapacitor URL/port remains the same.

We also have some subscriptions configured in our kapacitor.conf:

[[influxdb]]
  ...
  [influxdb.subscriptions]
    telegraf = ["autogen"]
  ...

Which causes Kapacitor to create a new subscription every time the server is initialized, if subscription is not there. And it uses the current Kapacitor instance' ServerID to name a subscription. So on our InfluxDB (1.4.2) server it then looks like:

> show subscriptions
name: telegraf
retention_policy name                                           mode destinations
---------------- ----                                           ---- ------------
autogen          kapacitor-c11618fc-5af5-4a1c-bb81-a66d19f0fa23 ANY  [http://my.kapacitor.hostname:9092]

The problem is that the mentioned ServerID attribute changes every time the server is initialized (container is re-run), which makes it to create a new subscription as this time the name for a subscription would not have a match within the existing subscriptions. So we end up with a new subscription added, something like:

> show subscriptions
name: telegraf
retention_policy name                                           mode destinations
---------------- ----                                           ---- ------------
autogen          kapacitor-c11618fc-5af5-4a1c-bb81-a66d19f0fa23 ANY  [http://my.kapacitor.hostname:9092]
autogen          kapacitor-4399ee14-a4aa-4c11-9d55-1f3c99ef6223 ANY  [http://my.kapacitor.hostname:9092]

That wouldn't be a problem, considering that Kapacitor would accept a subscription data only if URL matches and name matches (something I derived from https://github.com/influxdata/kapacitor/issues/349#issuecomment-197908046). But the problem is that with 2 existing subscriptions as above, I then receive a duplicated data when using a stream() node - each coming from the existing subscription. And if I drop one subscription, I stop getting the duplicated data. If I instead re-run the Kapacitor again it'll generate a new subscription and I'll be receiving the data thrice, and so on. So it looks like a subscription name (which includes ServerID) is not accounted when Kapacitor decides whether to accept a data or not.

What would be a suggested solution to run Kapacitor in such environment and scenario?

I can create a container pre-run script which wipes out not used subscriptions, but it looks more like a workaround. Alternatively, I can remove subscription section in kapacitor.conf and pre-create a subscription manually - but this is a workaround too. Even if Kapacitor would account the name (comprised of ServerID) of a subscription, I would still end up with tens of not used subscriptions accumulated over the time (considering we re-deploy the container often) - what would be a solution for Kapacitor then?

stanislav-zaprudskiy commented 6 years ago

I can create a container pre-run script which wipes out not used subscriptions, but it looks more like a workaround. Alternatively, I can remove subscription section in kapacitor.conf and pre-create a subscription manually - but this is a workaround too. Even if Kapacitor would account the name (comprised of ServerID) of a subscription, I would still end up with tens of not used subscriptions accumulated over the time (considering we re-deploy the container often) - what would be a solution for Kapacitor then?

I found this - https://github.com/influxdata/kapacitor/issues/870, which discusses the problem of cleaning up obsolete subscriptions. Though no one mentioned the issue I'm running into with duplicated data receive.