Unexpected newline error

[calebhailey]

I'm having some trouble sending metrics to InfluxDB, and getting an error like this:

{"component":"pipelined","environment":"default","handler":"influxdb","level":"info","msg":"event pipe handler executed","organization":"default","output":"/usr/bin/sensu-influxdb-handler: line 7: syntax error: unexpected newline\n","status":2,"time":"2018-09-05T10:17:23Z"}

Here's my check configuration (using the monitoring-plugins.org / Nagios check_http plugin packaged up as a Sensu 2.0 asset):

{
    "type": "Check",
    "spec": {
        "organization": "default",
        "environment": "default",
        "name": "check_sensu_io",
        "command": "check_http -H sensu.io -N",
        "runtime_assets": ["check_http_v0.1"],
        "publish": true,
        "interval": 10,
        "subscriptions": ["docker"],
        "output_metric_format": "nagios_perfdata",
        "output_metric_handlers": ["influxdb"]
    }
}

It produces output like this:

$ sensuctl event info 8c8a5e396ef8 check_sensu_io --format json
{
  "timestamp": 1536142813,
  "entity": {
    "class": "agent",
    "deregister": true,
    "deregistration": {},
    "environment": "default",
    "id": "8c8a5e396ef8",
    "keepalive_timeout": 60,
    "last_seen": 1536139981,
    "organization": "default",
    "redact": [
      "password",
      "passwd",
      "pass",
      "api_key",
      "api_token",
      "access_key",
      "secret_key",
      "private_key",
      "secret"
    ],
    "subscriptions": [
      "docker",
      "poller",
      "entity:8c8a5e396ef8"
    ],
    "system": {
      "hostname": "8c8a5e396ef8",
      "os": "linux",
      "platform": "alpine",
      "platform_family": "alpine",
      "platform_version": "3.8.0",
      "network": {
        "interfaces": [
          {
            "name": "lo",
            "addresses": [
              "127.0.0.1/8"
            ]
          },
          {
            "name": "tunl0",
            "addresses": null
          },
          {
            "name": "ip6tnl0",
            "addresses": null
          },
          {
            "name": "eth0",
            "mac": "02:42:ac:13:00:07",
            "addresses": [
              "172.19.0.7/16"
            ]
          }
        ]
      },
      "arch": "amd64"
    },
    "user": "agent"
  },
  "check": {
    "command": "check_http -H sensu.io -N",
    "environment": "default",
    "handlers": [],
    "high_flap_threshold": 0,
    "interval": 10,
    "low_flap_threshold": 0,
    "name": "check_sensu_io",
    "organization": "default",
    "publish": true,
    "runtime_assets": [
      "check_http_v0.1"
    ],
    "subscriptions": [
      "docker"
    ],
    "proxy_entity_id": "",
    "check_hooks": null,
    "stdin": false,
    "subdue": null,
    "ttl": 0,
    "timeout": 0,
    "round_robin": false,
    "duration": 0.2294587,
    "executed": 1536142812,
    "history": [
      {
        "status": 0,
        "executed": 1536142613
      },
      {
        "status": 0,
        "executed": 1536142623
      },
      {
        "status": 0,
        "executed": 1536142633
      },
      {
        "status": 0,
        "executed": 1536142643
      },
      {
        "status": 0,
        "executed": 1536142653
      },
      {
        "status": 0,
        "executed": 1536142663
      },
      {
        "status": 0,
        "executed": 1536142673
      },
      {
        "status": 0,
        "executed": 1536142683
      },
      {
        "status": 0,
        "executed": 1536142693
      },
      {
        "status": 0,
        "executed": 1536142703
      },
      {
        "status": 0,
        "executed": 1536142713
      },
      {
        "status": 0,
        "executed": 1536142723
      },
      {
        "status": 0,
        "executed": 1536142733
      },
      {
        "status": 0,
        "executed": 1536142743
      },
      {
        "status": 0,
        "executed": 1536142753
      },
      {
        "status": 0,
        "executed": 1536142763
      },
      {
        "status": 0,
        "executed": 1536142773
      },
      {
        "status": 0,
        "executed": 1536142783
      },
      {
        "status": 0,
        "executed": 1536142793
      },
      {
        "status": 0,
        "executed": 1536142802
      },
      {
        "status": 0,
        "executed": 1536142812
      }
    ],
    "issued": 1536142812,
    "output": "HTTP OK: HTTP/1.1 301 Moved Permanently - 280 bytes in 0.227 second response time |time=0.226747s;;;0.000000;10.000000 size=280B;;;0\n",
    "state": "passing",
    "status": 0,
    "total_state_change": 0,
    "last_ok": 1536142813,
    "occurrences": 1,
    "occurrences_watermark": 1,
    "output_metric_format": "nagios_perfdata",
    "output_metric_handlers": [
      "influxdb"
    ],
    "env_vars": null
  },
  "metrics": {
    "handlers": [
      "influxdb"
    ],
    "points": [
      {
        "name": "time",
        "value": 0.226747,
        "timestamp": 1536142812,
        "tags": []
      },
      {
        "name": "size",
        "value": 280,
        "timestamp": 1536142812,
        "tags": []
      }
    ]
  }
}

Here's my handler configuration:

{
    "type": "Handler",
    "spec": {
        "name": "influxdb",
        "type": "pipe",
        "command": "/usr/bin/sensu-influxdb-handler --addr 'http://influxdb:8086' -d 'sensu' -u 'admin' -p 'admin'",
        "timeout": 10,
        "filters": [
            "has_metrics"
        ],
        "organization": "default",
        "environment": "default"
    }
}

I'm sure I'm doing something wrong here, but I haven't been able to figure it out!

Thanks in advance for your help. :)

[nikkictl]

Thanks for filing this issue in such detail Caleb, it’s super helpful. Before I dig in a little further, I suspect I may just have to bump the sensu-go sha used to validate the event. We recently fixed a bug in sensu-go that allows multi line nagios perf metric data, and the event validation function probably is not updated in this repo.

[nikkictl]

@calebhailey I bumped the sensu-go version in this project to address my initial concern above. When you get a chance, could you try v1.7 out and let me know if that resolves your issue? If not, I'll definitely dig deeper into this!

[calebhailey]

OMG. Please label this issue "PEBKAC" and close it. So much facepalm. 🤦‍♂️ I'm too embarrassed to even explain what I did wrong, so I'll just say it was 100% operator error, haha.

[nikkictl]

Haha well regardless of the solution, thanks for filing it! You'll have to spill the beans to me some other time 😉