Running `sensu-aggregate-check` fails with cryptic message.

[madelaney]

All,

When I try to run sensu-aggregate-check, against a local instance of sensu-go-backend (5.18.1-9930), I'm getting a cryptic error message:

 sudo -H /opt/sensu-plugins-ruby/embedded/bin/sensu-aggregate-check --api-port 3080 --check-labels='aggregate=stability'
Error: invalid character 'C' looking for beginning of value
Usage:
  sensu-aggregate-check [flags]

Flags:
  -H, --api-host string        Sensu Go Backend API Host (e.g. 'sensu-backend.example.com') (default "127.0.0.1")
  -P, --api-pass string        Sensu Go Backend API User (default "P@ssw0rd!")
  -p, --api-port string        Sensu Go Backend API Port (e.g. 4242) (default "8080")
  -u, --api-user string        Sensu Go Backend API User (default "admin")
  -l, --check-labels string    Sensu Go Event Check Labels to filter by (e.g. 'aggregate=foo')
  -C, --crit-count int         Critical threshold - count of Events in critical state
  -c, --crit-percent int       Critical threshold - % of Events in critical state
  -e, --entity-labels string   Sensu Go Event Entity Labels to filter by (e.g. 'aggregate=foo,app=bar')
  -h, --help                   help for sensu-aggregate-check
  -n, --namespaces string      Comma-delimited list of Sensu Go Namespaces to query for Events (e.g. 'us-east-1,us-west-2') (default "default")
  -W, --warn-count int         Warning threshold - count of Events in warning state
  -w, --warn-percent int       Warning threshold - % of Events in warning state

error: invalid character 'C' looking for beginning of value
Exit 1

I can sign into the api using curl sudo curl -k --user admin:"...." https://127.0.0.1:3080/auth so I know my --api-port is ready but I don't know what is wrong here.

• Mike D.

[madelaney]

I have the follow asset configured:

sudo -H sensuctl asset info sensu/sensu-aggregate-check
type: Asset
api_version: core/v2
metadata:
  annotations:
    io.sensu.bonsai.api_url: https://bonsai.sensu.io/api/v1/assets/sensu/sensu-aggregate-check
    io.sensu.bonsai.name: sensu-aggregate-check
    io.sensu.bonsai.namespace: sensu
    io.sensu.bonsai.tags: aggregate, check
    io.sensu.bonsai.tier: Community
    io.sensu.bonsai.url: https://bonsai.sensu.io/assets/sensu/sensu-aggregate-check
    io.sensu.bonsai.version: 0.0.7
  name: sensu/sensu-aggregate-check
  namespace: default
spec:
  builds:
  - filters:
    - entity.system.os == 'windows'
    - entity.system.arch == 'amd64'
    headers: null
    sha512: dbff92a09a539320e28c45acc64ab6f0bb3c9f1f328910ad1236fd32e36055eb83f5ed42d353aa138bc1d8d1b736e51b50b331a8d36b10957d8301a28a5b0936
    url: https://assets.bonsai.sensu.io/c0e2af9f04f1e4089eb6c9d639ad8f856fca2316/sensu-aggregate-check_0.0.7_windows_amd64.tar.gz
  - filters:
    - entity.system.os == 'darwin'
    - entity.system.arch == '386'
    headers: null
    sha512: 71aeec73b9169a3b266d05c0d8bf0e7f4c2ac468dbb147681b7f7d4c7574ff43891e9c4a993c30434827c06973f80162f40adfa0e849bd17e7480bf7d2e5d4f0
    url: https://assets.bonsai.sensu.io/c0e2af9f04f1e4089eb6c9d639ad8f856fca2316/sensu-aggregate-check_0.0.7_darwin_386.tar.gz
  - filters:
    - entity.system.os == 'darwin'
    - entity.system.arch == 'amd64'
    headers: null
    sha512: 79002cba5050ba1b450fa38e1192bc52ae12267feda3879a22559a2794a6e20aeb2ed0e7f944265c811e7f17d2993d36e2680f4e9a80e4b6bc4c5687ce015f5a
    url: https://assets.bonsai.sensu.io/c0e2af9f04f1e4089eb6c9d639ad8f856fca2316/sensu-aggregate-check_0.0.7_darwin_amd64.tar.gz
  - filters:
    - entity.system.os == 'linux'
    - entity.system.arch == 'armv7'
    headers: null
    sha512: e8f74ad22e168c5e7a35bd5e90efcff9dffe1519d417c53a1f3fea5a16cc616f49ee1c89ee9564fccc29cf66b1400e9e4f9b907bd755672d696c3b04a044b1e5
    url: https://assets.bonsai.sensu.io/c0e2af9f04f1e4089eb6c9d639ad8f856fca2316/sensu-aggregate-check_0.0.7_linux_armv7.tar.gz
  - filters:
    - entity.system.os == 'linux'
    - entity.system.arch == 'arm64'
    headers: null
    sha512: 46372aa32b003b4362b31c05613fed0eea2bdca9b0502b9e057c243c2f3e56601fab5dbf9a0eb87e017c05327c42d41268fc2b7838a77543a8825e81ef108f03
    url: https://assets.bonsai.sensu.io/c0e2af9f04f1e4089eb6c9d639ad8f856fca2316/sensu-aggregate-check_0.0.7_linux_arm64.tar.gz
  - filters:
    - entity.system.os == 'linux'
    - entity.system.arch == '386'
    headers: null
    sha512: 7004791dbb1d2762363b023d14565eea6dbcce4214e2ca57b8c628cdbee6860835bf74b8119fb09576e0eb63f544e83c0f0e24dbc2a5ccc2c6ee7413420736d8
    url: https://assets.bonsai.sensu.io/c0e2af9f04f1e4089eb6c9d639ad8f856fca2316/sensu-aggregate-check_0.0.7_linux_386.tar.gz
  - filters:
    - entity.system.os == 'linux'
    - entity.system.arch == 'amd64'
    headers: null
    sha512: de6f2f5901b926591749dcd67b86283070867a1c2951beb1421a528ae97269ec7a6e73a35fbbbe6f1e06e8b9147b3aec234254f2208ed193ad465c1da1d92bb0
    url: https://assets.bonsai.sensu.io/c0e2af9f04f1e4089eb6c9d639ad8f856fca2316/sensu-aggregate-check_0.0.7_linux_amd64.tar.gz
  filters: null
  headers: null

[derekgroh]

Have you tried adding --warn-percent=75 --crit-percent=50 to your check command to see if it resolve the error?

[madelaney]

@derekgroh , I had not before but appending --warn-percent=75 --crit-percent=50 resulted in no change in behavior.

sudo -H /opt/sensu-plugins-ruby/embedded/bin/sensu-aggregate-check --api-port=3080 --check-labels='aggregate=stability' --warn-percent=75 --crit-percent=50
Error: invalid character 'C' looking for beginning of value
Usage:
  sensu-aggregate-check [flags]

Flags:
  -H, --api-host string        Sensu Go Backend API Host (e.g. 'sensu-backend.example.com') (default "127.0.0.1")
  -P, --api-pass string        Sensu Go Backend API User (default "P@ssw0rd!")
  -p, --api-port string        Sensu Go Backend API Port (e.g. 4242) (default "8080")
  -u, --api-user string        Sensu Go Backend API User (default "admin")
  -l, --check-labels string    Sensu Go Event Check Labels to filter by (e.g. 'aggregate=foo')
  -C, --crit-count int         Critical threshold - count of Events in critical state
  -c, --crit-percent int       Critical threshold - % of Events in critical state
  -e, --entity-labels string   Sensu Go Event Entity Labels to filter by (e.g. 'aggregate=foo,app=bar')
  -h, --help                   help for sensu-aggregate-check
  -n, --namespaces string      Comma-delimited list of Sensu Go Namespaces to query for Events (e.g. 'us-east-1,us-west-2') (default "default")
  -W, --warn-count int         Warning threshold - count of Events in warning state
  -w, --warn-percent int       Warning threshold - % of Events in warning state

error: invalid character 'C' looking for beginning of value
Exit 1

[asachs01]

FWIW, I'm seeing the same thing in a scenario that previously used to work for me. Maybe @jspaleta or @nixwiz might have an idea of why it's crapping out

[nixwiz]

@madelaney I'm assuming your backend API port is using TLS?

[madelaney]

@nixwiz , yes it is.

[madelaney]

I tested a configuration without TLS configured in backend.yaml, my command worked fine. I removed the keys cert-file, key-file, and trusted-ca-file.

[madelaney]

Are there any build instructions for how to build this?

[nixwiz]

@madelaney I've not submitted a PR for this yet, but I have a branch in my fork for adding secure connections (as well as using an API Key instead of username/password). If you have a golang 1.14 version compiler installed you should be able to clone my repo and checkout this branch and do a go build. I'd love to have someone test it with real data.

[madelaney]

@nixwiz , I tested two configs with mixed results. When I try with the -i flag, it works as expected. However when I tried to pass in the trust ca file, I was not able to get it to work maybe it was an error on my side?

Insecure Mode

./sensu-aggregate-check --api-port 9080 --check-labels='aggregate=stability,app=jira' -s -i
Counters: {Entities:11 Checks:1 Ok:11 Warning:0 Critical:0 Unknown:0 Total:11}
Percent OK: 100
Everything is OK

Secure Mode

sudo cat /etc/sensu/backend.yml | grep trust
trusted-ca-file: "/etc/sensu/ssl/ca.crt"

sudo ./sensu-aggregate-check --api-port 9080 --check-labels='aggregate=stability,app=jira' -s -t /etc/sensu/ssl/ca.crt
Usage:
  sensu-aggregate-check [flags]
  sensu-aggregate-check [command]

Available Commands:
  help        Help about any command
  version     Print the version number of this plugin

Flags:
  -H, --api-host string          Sensu Go Backend API Host (e.g. 'sensu-backend.example.com') (default "127.0.0.1")
  -k, --api-key string           Sensu Go Backend API Key
  -P, --api-pass string          Sensu Go Backend API Password (default "P@ssw0rd!")
  -p, --api-port string          Sensu Go Backend API Port (e.g. 4242) (default "8080")
  -u, --api-user string          Sensu Go Backend API User (default "admin")
  -l, --check-labels string      Sensu Go Event Check Labels to filter by (e.g. 'aggregate=foo')
  -C, --crit-count int           Critical threshold - count of Events in warning state
  -c, --crit-percent int         Critical threshold - % of Events in warning state
  -e, --entity-labels string     Sensu Go Event Entity Labels to filter by (e.g. 'aggregate=foo,app=bar')
  -h, --help                     help for sensu-aggregate-check
  -i, --insecure-skip-verify     skip TLS certificate verification (not recommended!)
  -n, --namespaces string        Comma-delimited list of Sensu Go Namespaces to query for Events (e.g. 'us-east-1,us-west-2') (default "default")
  -s, --secure                   Use TLS connection to API
  -t, --trusted-ca-file string   TLS CA certificate bundle in PEM format
  -W, --warn-count int           Warning threshold - count of Events in warning state
  -w, --warn-percent int         Warning threshold - % of Events in warning state

Use "sensu-aggregate-check [command] --help" for more information about a command.

Error executing sensu-aggregate-check: error executing check: Get "https://127.0.0.1:9080/auth": x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs
Exit 3

[nixwiz]

To me that says that the certificate you are asking to be trusted by using that CA does not have the default host name (127.0.0.1) as the CN or the certificate or as a SAN. Therefore it won't accept it. If you were to specify the name that matches with -H it should work.

[madelaney]

Yep, if I pass in the -H flag it works:

sudo ./sensu-aggregate-check --api-port 9080 --check-labels='aggregate=stability,app=jira' -s -t /etc/sensu/ssl/ca.crt -H <fqdn here>
Counters: {Entities:11 Checks:1 Ok:11 Warning:0 Critical:0 Unknown:0 Total:11}
Percent OK: 100
Everything is OK

[nixwiz]

Awesome. I'm going to submit my PR including these changes and hopefully get this rolled into a release soon.

[madelaney]

Awesome, thanks for your effort in this @nixwiz

[edanidzerda]

In our environment we had to switch to use API keys to solve random errors when using the aggregate check API. When I added debugging we saw the Sensu API returning

{"Code":4,"Message":"unauthorized to perform action"}

Using an API key solved this problem for us. FYI it seemed worse when we ran multiple aggregate queries at the same check frequency.