"Too many requests" error when using sensu-influxdb-handler

[vinkris]

I have a sensu-backend configuration as follows in Terraform:

resource "sensu_handler" "influxdb_handler" {
  name = "influxdb_handler"

  type    = "pipe"
  command = "sensu-influxdb-handler -d sensu"
  timeout = 10

  env_vars = {
    INFLUXDB_ADDR = "http://influxdb.${var.aws_region}.${var.domain_name}:9096"
  }

  runtime_assets = [
    "influxdb_handler_asset",
  ]

  filters = [
    "has_metrics",
  ]
}

resource "sensu_asset" "influxdb_handler_asset" {
  name   = "influxdb_handler_asset"
  sha512 = "612c6ff9928841090c4d23bf20aaf7558e4eed8977a848cf9e2899bb13a13e7540bac2b63e324f39d9b1257bb479676bc155b24e21bf93c722b812b0f15cb3bd"
  url    = "https://github.com/sensu/sensu-influxdb-handler/releases/download/3.1.2/sensu-influxdb-handler_3.1.2_linux_amd64.tar.gz"
}

I constantly keep seeing this error in sensu-backend logs:

{
    "assets": [
        "influxdb_handler_asset"
    ],
    "component": "pipelined",
    "handler": "influxdb_handler",
    "level": "info",
    "msg": "event pipe handler executed",
    "namespace": "default",
    "output": "Error: \"Too Many Requests\"\nUsage:\n  sensu-influxdb-handler [flags]\n\nFlags:\n  -a, --addr string            the address of the influxdb server, should be of the form 'http://host:port', defaults to 'http://localhost:8086' or value of INFLUXDB_ADDR env variable (default \"http://influxdb-relay.us-east-2.ess.aws.cudaops.com:9096\")\n  -d, --db-name string         the influxdb to send metrics to\n  -h, --help                   help for sensu-influxdb-handler\n  -i, --insecure-skip-verify   if true, the influx client skips https certificate verification\n  -p, --password string        the password for the given db, defaults to value of INFLUXDB_PASS env variable\n      --precision string       the precision value of the metric (default \"s\")\n  -u, --username string        the username for the given db, defaults to value of INFLUXDB_USER env variable\n\nerror: \"Too Many Requests\"\n",
    "status": 1,
    "time": "2019-06-05T15:16:26Z"
}

Should I be worried about this? Anything I can do to improve the configuration? Please advise. Thanks.

[nikkictl]

Hey @vinkris, sorry you're running into this issue! Is the influxdb instance running on the same node as your sensu backend? If so, is it running properly after deploying? If not, can you ping the resulting influxdb address from the sensu backend node? I was wondering if you could provide a little additional context here regarding your Sensu resources, as well. How many checks are being handled by influxdb_handler? How often are those checks being executed?

[vinkris]

Hi @nikkiki, thank you so much for responding promptly on this. We were using a fork of influxdb-relay, and turns out that had a default burst-limit of 10, which is too less for the traffic.

We have close to 300 servers sending metrics every minute, and after I tuned the burst-limit of influxdb-relay things look good now.

Sorry about the false alarm. Thanks again for taking time to look into this.