"possible authentication failure. wrong credentials?","user":"sensu"

[ismc]

Not sure if this is an issue with the playbooks or my intelligence, but I've exhausted all of the docs that I can find. I am using ansible-sensu to install Sensu on a CentOS-7 host with the following playbook:

- hosts: sensu_masters
  become: yes

  tasks:
    - name: Install EPEL
      yum:
        name: epel-release

    - name: Install sensu
      include_role:
        name: sensu
      vars:
        sensu_master: true
        sensu_include_dashboard: true
        redis_server: true
        sensu_deploy_redis: true
        rabbitmq_server: true
        rabbitmq: true
        sensu_deploy_rabbitmq: true
        sensu_include_plugins: true
        dynamic_data_store: "{{ playbook_dir }}/sensu_data/store"
        static_data_store: "{{ playbook_dir}}/sensu_data/static"
        rabbitmq_host: 192.168.12.16
        redis_host: 192.168.12.16
        sensu_api_host: 192.168.12.16
        rabbitmq_sensu_user_name: sensu
        rabbitmq_sensu_password: sensu
        sensu_api_user_name: sensu
        sensu_api_password: sensu

    - name: Allow ports through firewall
      firewalld:
        port: "{{ item }}"
        permanent: true
        state: enabled
        immediate: yes
      with_items:
        - 3000/tcp
        - 5671/tcp
        - 6379/tcp
        - 4567/tcp

Everything installs ok and I can log into the dashboard. However, I get a Datacenter sensu-81 returned: 500 Internal Server Error and I see

{"timestamp":"2017-10-14T12:51:15.761755-0400","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-10-14T12:51:15.761841-0400","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}
{"timestamp":"2017-10-14T12:51:16.290375-0400","level":"info","message":"api response","request":{"remote_address":"192.168.12.16","user_agent":"Go-http-client/1.1","method":"GET","uri":"/stashes","query_string":null,"body":""},"status":500,"content_length":59,"time":0.0}

in /var/log/sensu/sensu-api.log. It seems obvious that some component us using the wrong creds to access the API, but I do not know which those are. Can someone point me to the doc that I am missing to make this work?

Thanks,

Steven.

[jcoetsie]

I have the identical same issue but on ubuntu.

With an almost default setup:

---

- hosts: all
  user: ubuntu
  become: yes
  roles:
    - { role: cmacrae.sensu }

sensu-master host_vars:

sensu_master: true
sensu_include_dashboard: true
rabbitmq_server: true
redis_server: true

[jaredledvina]

Hey @ismc and @jcoetsie,

So sorry for the delay in responding, are you still hitting this issue?

[hgfranco]

@jaredledvina. I'm experiencing the same problem. Looks like it's a CentOS/RedHat 7 issue since running a similar ansible setup on CentOS 6 yields no errors.

Cent6: rabbitmq - 2.6.1 sensu - 1.2.0

Cent7: rabbitmq - 3.3.5 sensu - 1.2.0

[ismc]

I have not had success either.

On Tue, Jan 2, 2018 at 1:25 PM Henry Franco notifications@github.com wrote:

@jaredledvina https://github.com/jaredledvina. I'm experiencing the same problem. Looks like it's a CentOS/RedHat 7 issue since running a similar ansible setup on CentOS 6 yields no errors.

Cent6: rabbitmq - 2.6.1 sensu - 1.2.0

Cent7: rabbitmq - 3.3.5 sensu - 1.2.0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sensu/sensu-ansible/issues/102#issuecomment-354839141, or mute the thread https://github.com/notifications/unsubscribe-auth/AJ5XO_fzZ1SuhuWQ9NiCNBt1jHqRjqLAks5tGnSHgaJpZM4P5c68 .

-- Steven Carter Principal Solutions Architect, Ansible scarter@redhat.com, (919) 909-1691

[jaredledvina]

Hey @hgfranco / @ismc,

Odd, can you comment here with your /var/log/sensu-api.log with the error's you're seeing and with the corresponding RabbitMQ log? I think RabbitMQ will also have some information logged that will help. Also, are you using this ansible role to manage/configure RabbitMQ with SSL certificates? And finally, when you run the ansible task, is this task finishing successfully: https://github.com/sensu/sensu-ansible/blob/master/tasks/rabbit.yml#L48-L58 ?

[jaredledvina]

Also, with the configured RabbitMQ credentials from the host running RabbitMQ are you able to use the following command successfully?

curl -i -u sensu:$your_rabbitmq_sensu_password_here http://localhost:15672/api/whoami

I believe, you should see something like the following:

❯ curl -i -u sensu:super-secret-password http://localhost:15672/api/whoami
HTTP/1.1 200 OK
cache-control: no-cache
content-length: 39
content-type: application/json
date: Tue, 02 Jan 2018 18:38:16 GMT
server: Cowboy
vary: accept, accept-encoding, origin

{"name":"sensu","tags":"administrator"}

[hgfranco]

@jaredledvina here's what sensu-api.log is spitting out

{"timestamp":"2018-01-02T17:19:34.017791-0500","level":"info","message":"api response","request":{"request_id":"f5ed3b47-8995-4fe2-b8b3-454f11d1ce50","remote_address":"127.0.0.1","user_agent":"Go-http-client/1.1","method":"GET","uri":"/stashes","query_string":null,"body":""},"status":500,"content_length":59,"time":0.0}
{"timestamp":"2018-01-02T17:19:35.762277-0500","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2018-01-02T17:19:35.762440-0500","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}

Here's what my rabbitmq log is show me:

=ERROR REPORT==== 2-Jan-2018::17:21:57 ===
error on AMQP connection <0.10719.0>:
{ssl_upgrade_failure,
    {{function_clause,
         [{tls_v1,enum_to_oid,[28],[{file,"tls_v1.erl"},{line,404}]},
          {ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,
              [{file,"ssl_handshake.erl"},{line,1657}]},
          {ssl_handshake,'-dec_hello_extensions/2-blc$^1/1-0-',1,
              [{file,"ssl_handshake.erl"},{line,1657}]},
          {ssl_handshake,dec_hello_extensions,2,
              [{file,"ssl_handshake.erl"},{line,1657}]},
          {tls_handshake,decode_handshake,3,
              [{file,"tls_handshake.erl"},{line,182}]},
          {tls_handshake,get_tls_handshake_aux,3,
              [{file,"tls_handshake.erl"},{line,153}]},
          {tls_connection,next_state,4,
              [{file,"tls_connection.erl"},{line,454}]},
          {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,505}]}]},
     {gen_fsm,sync_send_all_state_event,[<0.10720.0>,{start,5000},infinity]}}}

I've verified that rabbitmq-server is running and everything looks ok.

I'm not using the task you linked to but my tasks is pretty similiar to what you have:

- block:
  - name: Manage vhost
    rabbitmq_vhost:
      name: "{{ rabbitmq_vhost }}"
      state: present

  - name: Manage user
    rabbitmq_user:
      state: present
      user: "{{ rabbitmq_user }}"
      password: "{{ rabbitmq_password }}"
      vhost: "{{ rabbitmq_vhost }}"
      configure_priv: .*
      read_priv: .*
      write_priv: .*
  when:
    - rabbitmq_user is defined
    - rabbitmq_password is defined
    - rabbitmq_vhost is defined

That curl command isn't working for me: # curl -i -u sensu:$rabbitmq-sensu-password http://localhost:15672/api/whoami curl: (7) Failed to connect to ::1: Network is unreachable

[jaredledvina]

Hey @hgfranco,

Nice, that's useful, from that RabbitMQ log messages, it seems that there's definitely something incorrect with the way RabbitMQ is configured. I haven't personally seen the ssl_upgrade_failure message before.

Which version of RabbitMQ and Erlang are installed on that host?

From https://www.rabbitmq.com/troubleshooting-ssl.html:

Entries containing {ssl_upgrade_failure, ... certify ...} This error is related to client verification. The client is presenting an invalid certificate or no certificate. If the ssl_options has the verify option set to verify_peer then try using the value verify_none temporarily. Ensure that the client certificate has been generated correctly, and that the client is presenting the correct certificate.

Based on that, can you paste here the /etc/sensu/conf.d/rabbitmq.json config (please redact anything sensitive :) ) and then, give the above recommendation a shot? If that works, could you try the steps outlined under "Check Keys and Certificates with OpenSSL" on that same page? My current theory is that somehow your client certificates aren't valid/trusted by RabbitMQ but, I'm not entirely sure.

[jaredledvina]

Hey @hgfranco, Were you ever able to get to the bottom of this?

[romainrbr]

Getting the same issue with CentOS 7. According to RabbitMQ's FAQ, we should use Erlang >=19.3 for TLS.

Using those RPM seems to fix the issue : https://github.com/rabbitmq/erlang-rpm Pushing a PR to fix it

[jaredledvina]

Hey @romainrbr - Thanks for the PR!

@hgfranco - Can you try to upgrade Erlang, this role is updated to support that now, let me know if that resolves your issue.

[jaredledvina]

With https://github.com/sensu/sensu-ansible/pull/130 merged in, the updated version of Erland/RabbitMQ should resolve the issues here. I'll be cutting the 2.0.0 release of this role today/tomorrow as well which will officially have that change. If you'd like to use it before that feel free to pull the role directly from Github.

I think this is clear to close out but if I'm missing anything just let me know, happy to continue troubleshooting. I also invite anyone to swing by #help (or #ansible) in the Sensu Community Slack room for more real-time troubleshooting.

[krithi2]

Hello All,

I seem to have a problem in the Uchiwa dashbord . It gives me an error "cannot load the uchiwa configuration" .

Using the puppet forge modules for Uchiwa Sensu Rabbitmq Redis .

Below are the log files :+1:

● uchiwa.service - LSB: Uchiwa, a Sensu dashboard.
   Loaded: loaded (/etc/rc.d/init.d/uchiwa; bad; vendor preset: disabled)
   Active: active (exited) since Wed 2018-02-14 04:08:47 EST; 1h 47min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 5051 ExecStop=/etc/rc.d/init.d/uchiwa stop (code=exited, status=0/SUCCESS)
  Process: 5061 ExecStart=/etc/rc.d/init.d/uchiwa start (code=exited, status=0/SUCCESS)

Feb 14 04:08:47 sensu.vm.com systemd[1]: Starting LSB: Uchiwa, a Sensu dashboard....
Feb 14 04:08:47 sensu.vm.com uchiwa[5061]: uchiwa started.
Feb 14 04:08:47 sensu.vm.com systemd[1]: Started LSB: Uchiwa, a Sensu dashboard..
Feb 14 04:08:47 sensu.vm.com su[5064]: (to uchiwa) root on none
[vagrant@sensu ~]$ 

[root@sensu conf.d]# tail -f /var/log/uchiwa.log 
{"timestamp":"2018-02-14T05:42:23.13277836-05:00","level":"info","message":"Updating the datacenter sensu"}
{"timestamp":"2018-02-14T05:42:28.133447015-05:00","level":"warn","message":"GET https://127.0.0.1:4567/sensu/stashes returned: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}
{"timestamp":"2018-02-14T05:42:28.133561144-05:00","level":"warn","message":"Connection failed to the datacenter sensu"}
{"timestamp":"2018-02-14T05:42:28.13361067-05:00","level":"info","message":"Updating the datacenter sensu"}
{"timestamp":"2018-02-14T05:42:33.134123579-05:00","level":"warn","message":"GET https://127.0.0.1:4567/sensu/stashes returned: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}
{"timestamp":"2018-02-14T05:42:33.134260766-05:00","level":"warn","message":"Connection failed to the datacenter sensu"}
{"timestamp":"2018-02-14T05:42:33.134306015-05:00","level":"info","message":"Updating the datacenter sensu"}
{"timestamp":"2018-02-14T05:42:38.134835092-05:00","level":"warn","message":"GET https://127.0.0.1:4567/sensu/stashes returned: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}
{"timestamp":"2018-02-14T05:42:38.134939244-05:00","level":"warn","message":"Connection failed to the datacenter sensu"}
{"timestamp":"2018-02-14T05:42:38.134982673-05:00","level":"info","message":"Updating the datacenter sensu"}

sensu-api.service - sensu api
   Loaded: loaded (/usr/lib/systemd/system/sensu-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-02-14 05:53:39 EST; 5min ago
 Main PID: 12984 (sensu-api)
   CGroup: /system.slice/sensu-api.service
           └─12984 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-api -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-api.pid -l /var/log/sensu/sensu-...

Feb 14 05:53:39 sensu.vm.com systemd[1]: Started sensu api.
Feb 14 05:53:39 sensu.vm.com systemd[1]: Starting sensu api...
[vagrant@sensu ~]$ 

[root@sensu conf.d]# tail -f /var/log/sensu/sensu-api.log
{"timestamp":"2018-02-14T05:41:25.540447-0500","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2018-02-14T05:41:25.540922-0500","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}
{"timestamp":"2018-02-14T05:41:45.539946-0500","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2018-02-14T05:41:45.540261-0500","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}
{"timestamp":"2018-02-14T05:42:05.539962-0500","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2018-02-14T05:42:05.540192-0500","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}
{"timestamp":"2018-02-14T05:42:25.540590-0500","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2018-02-14T05:42:25.540669-0500","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}
{"timestamp":"2018-02-14T05:42:45.542256-0500","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2018-02-14T05:42:45.542469-0500","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","u

sensu-client.service - sensu client
   Loaded: loaded (/usr/lib/systemd/system/sensu-client.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-02-14 05:53:41 EST; 3min 51s ago
 Main PID: 13001 (sensu-client)
   CGroup: /system.slice/sensu-client.service
           └─13001 /opt/sensu/embedded/bin/ruby /opt/sensu/bin/sensu-client -c /etc/sensu/config.json -d /etc/sensu/conf.d -e /etc/sensu/extensions -p /var/run/sensu/sensu-client.pid -l /var/log/sensu/...

Feb 14 05:53:41 sensu.vm.com systemd[1]: Started sensu client.
Feb 14 05:53:41 sensu.vm.com systemd[1]: Starting sensu client...

[root@sensu conf.d]# tail -f /var/log/sensu/sensu-client.log
{"timestamp":"2018-02-14T05:41:59.764609-0500","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2018-02-14T05:41:59.764693-0500","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}
{"timestamp":"2018-02-14T05:42:09.765490-0500","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2018-02-14T05:42:09.765577-0500","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}
{"timestamp":"2018-02-14T05:42:21.768444-0500","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2018-02-14T05:42:21.768687-0500","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}
{"timestamp":"2018-02-14T05:42:35.769374-0500","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2018-02-14T05:42:35.769622-0500","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}
{"timestamp":"2018-02-14T05:42:51.769198-0500","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2018-02-14T05:42:51.769386-0500","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}

[root@sensu conf.d]# curl -i -u sur:sur http://localhost:15672/api/whoami
HTTP/1.1 200 OK
Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
Date: Wed, 14 Feb 2018 10:43:37 GMT
Content-Type: application/json
Content-Length: 83
Cache-Control: no-cache

{"name":"sur","tags":"administrator","auth_backend":"rabbit_auth_backend_internal"}[root@sensu conf.d]# 

[root@sensu conf.d]# tail -f /var/log/rabbitmq/rabbit\@sensu.log 
accepting AMQP connection <0.359.0> ([::1]:47110 -> [::1]:5672)

=INFO REPORT==== 14-Feb-2018::05:43:45 ===
accepting AMQP connection <0.366.0> ([::1]:47184 -> [::1]:5672)

=INFO REPORT==== 14-Feb-2018::05:43:45 ===
accepting AMQP connection <0.369.0> (127.0.0.1:51612 -> 127.0.0.1:5672)

=INFO REPORT==== 14-Feb-2018::05:43:49 ===
accepting AMQP connection <0.372.0> ([::1]:47208 -> [::1]:5672)

COuld you please help me fix it . I ma trying to install this on an Centos7 machine.

[krithi2]

<Adding he config files as well :+1:

[root@sensu conf.d]# cat rabbitmq.json 
{
  "rabbitmq": {
    "port": 5672,
    "host": "127.0.0.1",
    "user": "sensu",
    "password": "sur",
    "vhost": "/sensu",
    "heartbeat": 30,
    "prefetch": 1
  }
}

[root@sensu conf.d]# cat redis.json 
{
  "redis": {
    "port": 6379,
    "host": "127.0.0.1",
    "reconnect_on_error": true,
    "db": 0,
    "auto_reconnect": true
  }
}

[root@sensu conf.d]# cat api.json 
{
  "api": {
    "port": 4567,
    "host": "127.0.0.1",
    "bind": "0.0.0.0"
  }
}

[root@sensu conf.d]# cat transport.json 
{
  "transport": {
    "name": "rabbitmq",
    "reconnect_on_error": true
  }
}

[root@sensu conf.d]# cd ..
[root@sensu sensu]# cat uchiwa.json 
{
"sensu": [
      {
        "name": "sensu",
        "host": "127.0.0.1",
        "ssl": true,
        "insecure": true,
        "port": 4567,
        "user": "sensu",
        "pass": "correct-horse-battery-staple",
        "path": "/sensu",
        "timeout": 5
      }
  ],
  "uchiwa": {
    "host": "0.0.0.0",
    "port": 3000,
    "user": "",
    "pass": "",
    "refresh": 5
  }
}

[krithi2]

Below is the list of Modules in puppetfile :+1:

#!/usr/bin/env ruby

forge "https://forgeapi.puppetlabs.com"

mod 'puppetlabs-motd', '1.7.0'
#mod 'puppetlabs-registry', '2.0.1'
mod 'puppetlabs-registry', '1.1.4'
mod 'puppetlabs-stdlib', '4.24.0'

#Redis Modules
mod 'dwerder-redis', '2.1.0'

mod 'puppet-rabbitmq', '8.1.0'
mod 'puppet-archive', '2.2.0'
mod 'garethr-erlang', '0.3.0'
mod 'stahnma-epel', '1.3.0'

#sensu MOdules
mod 'sensu-sensu', '2.50.0'
mod 'lwf-remote_file', '1.1.3'

#uchiwa Modules
mod 'yelp-uchiwa', '2.0.0'
mod 'puppetlabs-apt','4.5.1'

Please help me sort this out :(

Thanks , Krithi