search

sensu / sensu-puppet

Sensu Puppet module.
https://github.com/sensu/sensu-puppet
MIT License
165 stars 291 forks source link

Using rabbitmq_cluster works only the first time puppet runs #598

Closed b-0-b closed 7 years ago

b-0-b commented 7 years ago

Description of problem

If I remove /etc/sensu/conf.d/rabbitmq.json the following puppet run will work again but fail on the puppet run after.

Command used and debugging output

csoleimani commented 7 years ago

I can confirm that I am having the same issue. I tried using a forked module with johanek's pull request above, but I still get the same error: "Could not evaluate: no implicit conversion of String into Integer " after the rabbitmq.json file has already been generated. I am also on Debian Jesse.

cwjohnston commented 7 years ago

@b-0-b @csoleimani are you setting values for any rabbitmq_* parameters aside from rabbitmq_cluster ? I believe this error results from passing a value for one or more of the rabbitmq parameters which are obviated by the configuration passed to rabbitmq_cluster.

csoleimani commented 7 years ago

@cwjohnston , I checked a previous revision of our code and the only rabbitmq parameter that was being passed was rabbitmq_cluster - I can post what my code looked like if you'd like.

cwjohnston commented 7 years ago

@csoleimani please share your code. Myself and others have used rabbitmq_cluster parameter successfully so whatever code you can share may help us identify the delta.

csoleimani commented 7 years ago

Sure, here is some of the code below. I'm only pasting the important stuff rather than the whole manifest.

# Configure a sensu server
class profile::sensu::server::main {

  $adserver          = hiera('ad::server')
  $adport            = hiera('ad::port')
  $basedn            = hiera('ad::basedn')
  $binduser          = hiera('sensu::ad::binduser')
  $bindpass          = hiera('sensu::ad::bindpass')
  $client_version    = hiera('sensu::client::version')
  $dashboard_name    = upcase($::location)
  $groupbasedn       = hiera('ad::groupbasedn')
  $log_level         = hiera('sensu::log_level')
  $rabbitmq_password = hiera('sensu::rabbitmq_password')
  $rabbitmq_port     = '5672'
  $rabbitmq_vhost    = 'sensu'
  $rabbitmq_vip      = hiera('sensu::rabbitmq::vip')
  $rabbitmq_user     = 'sensu'
  $redis_host        = hiera('sensu::redis::vip')
  $redis_port        = hiera('redis::port')
  $sensu_api_port    = hiera('sensu::api::port')
  $sensu_fqdn        = hiera('sensu::fqdn')
  $sensu_cert        = $sensu_fqdn
  $security          = 'tls'
  $userattribute     = 'mail'
  $userbasedn        = hiera('ad::userbasedn')
  $rabbitmq_cluster  = $::app_tier ? {
    /(production|staging)/ => [
      {
        host               => "rabbitmq-${app_tier}-node01",
        user               => $rabbitmq_user,
        password           => $rabbitmq_password,
        port               => $rabbitmq_port,
        refetch            => '5',
        reconnect_on_error => true,
        vhost              => $rabbitmq_vhost
      },{
        host               => "rabbitmq-${app_tier}-node02",
        user               => $rabbitmq_user,
        password           => $rabbitmq_password,
        port               => $rabbitmq_port,
        prefetch           => '5',
        reconnect_on_error => true,
        vhost              => $rabbitmq_vhost
      },{
        host               => "$rabbitmq-${app_tier}-node03",
        user               => $rabbitmq_user,
        password           => $rabbitmq_password,
        port               => $rabbitmq_port,
        prefetch           => '5',
        reconnect_on_error => true,
        vhost              => $rabbitmq_vhost
      }
    ],
    'test' => [
    {
      host               => $rabbitmq_vip,
      user               => $rabbitmq_user,
      password           => $rabbitmq_password,
      port               => $rabbitmq_port,
      refetch            => '5',
      reconnect_on_error => true,
      vhost              => $rabbitmq_vhost
    }
    ],
    default => undef
  }
  $redis_db          = lookup('sensu::redis::db')

  class { '::sensu':
    enterprise                   => true,
    enterprise_version           => $::sensu::enterprise_version,
    enterprise_dashboard         => true,
    enterprise_dashboard_version => $::sensu::enterprise_dashboard_version,
    enterprise_dashboard_host    => 'localhost',
    enterprise_dashboard_port    => '3000',
    enterprise_dashboard_refresh => '5',
    enterprise_dashboard_ldap    => {
    'basedn'              => $basedn,
    'groupbasedn'         => $groupbasedn,
    'userbasedn'          => $userbasedn,
    'binduser'            => $binduser,
    'bindpass'            => $bindpass,
    'server'              => $adserver,
    'port'                => $adport,
    'roles'               => [{
    'name'    => 'operators',
    'members' => ['smart-people'],
    },
    {
    'name'     => 'guests',
    'members'  => ['look-but-dont- touch'],
    'readonly' => true,
    }
    ],
    'security'            => $security,
    'userattribute'       => $userattribute,
    'disablenestedgroups' => true,
    },
    enterprise_user              => 'REDACTED',
    enterprise_pass              => 'REDACTED',
    install_repo                 => true,
    log_level                    => $log_level,
    purge                        => {
      config => true,
    },
    rabbitmq_cluster             => $rabbitmq_cluster,
    redis_db                     => $redis_db,
    redis_host                   => $redis_host,
    version                      => $client_version,
  }
}
b-0-b commented 7 years ago

Hi,

I tried to do it all in hiera

sensu::rabbitmq_cluster:
  - port: 5671
    host: 192.168.113.9
    user: sensu
    password: password
    vhost: "sensu"
    ssl: {
      cert_chain_file: "/etc/sensu/ssl/cert.pem",
      private_key_file: "/etc/sensu/ssl/key.pem"
      }
  - port: 5671
    host: 192.168.113.10
    user: sensu
    password: password
    vhost: "sensu"
    ssl: {
      cert_chain_file: "/etc/sensu/ssl/cert.pem",
      private_key_file: "/etc/sensu/ssl/key.pem"
      }
  - port: 5671
    host: 192.168.113.11
    user: sensu
    password: password
    vhost: "sensu"
    ssl: {
      cert_chain_file: "/etc/sensu/ssl/cert.pem",
      private_key_file: "/etc/sensu/ssl/key.pem"
      }

This is what I was unsure about how to do. This will create /etc/sensu/conf.d/rabbitmq.json the way I want the first time puppet runs but fails at the next puppet run.

My puppet code is 

  class { '::sensu':
    init_stop_max_wait          => 120,
    log_level                   => 'info',
    manage_user                 => false,
    rabbitmq_reconnect_on_error => true,
    redis_reconnect_on_error    => true,
    redis_port                  => 6380,
    install_repo                => true,
    use_embedded_ruby           => true,
    manage_plugins_dir          => false,
    purge                       => {
      config     => true,
      mutators   => true,
      handlers   => true,
      extensions => true,
      plugins    => false,
    },
  }
kmcfate commented 7 years ago

I had the same problem, and confirmed that it was because i had rabbitmq_password set in hiera.

b-0-b commented 7 years ago

Right. Thanks for the info. I now found I have these in hiera as well

sensu::rabbitmq_password: eeCiephahx0kah5hoh4a
sensu::rabbitmq_ssl: true
csoleimani commented 7 years ago

I just tried hardcoding the password into the manifest and not doing a hiera lookup, but I'm still seeing the same issue. Could you please post what your code looks like, @kmcfate ? I just tried taking all of the hiera lookups out of my rabbitmq_cluster variable and it still is showing the same behavior.

cwjohnston commented 7 years ago

@csoleimani for your case i suspect that rabbitmq_reconnect_on_error is a contributing factor. Sensu defaults this to true, so removing the explicit definition from configuration shouldn't change behavior.

csoleimani commented 7 years ago

Thanks for the suggestion, @cwjohnston . I just tried removing it and unfortunately, removing that attribute didn't seem to get rid of the issue.

b-0-b commented 7 years ago

This works for me now after refactoring my puppet an hiera code.

manifest

  class foo::sensu
    class { '::sensu':
    init_stop_max_wait   => 120,
    log_level            => 'info',
    manage_user          => false,
    rabbitmq_cluster     => $rabbitmq_cluster,
    redis_password       => "redis_pass",
    redis_db             => 0,
    redis_auto_reconnect => true,
    redis_sentinels      => $sentinels,
    redis_master         => "sensu",
    install_repo         => true,
    use_embedded_ruby    => true,
    manage_plugins_dir   => false,
    purge                => {
      config     => true,
      mutators   => true,
      handlers   => true,
      extensions => true,
      plugins    => false,
    },
  }

hiera

foo::sensu::rabbitmq_cluster:
  - port: 5671
    host: 192.168.113.9
    user: sensu
    password: rabbitmq_pw
    vhost: "sensu"
    prefetch: 50
    heartbeat: 30
    ssl: {
      cert_chain_file: "/etc/sensu/rabbitmq_ssl_cert.pem",
      private_key_file: "/etc/sensu/rabbitmq_ssl_key.pem"
      }
  - port: 5671
    host: 192.168.113.10
    user: sensu
    password: rabbitmq_pw
    vhost: "sensu"
    prefetch: 50
    heartbeat: 30
    ssl: {
      cert_chain_file: "/etc/sensu/rabbitmq_ssl_cert.pem",
      private_key_file: "/etc/sensu/rabbitmq_ssl_key.pem"
      }
  - port: 5671
    host: 192.168.113.11
    user: sensu
    password: rabbitmq_pw
    vhost: "sensu"
    prefetch: 50
    heartbeat: 30
    ssl: {
      cert_chain_file: "/etc/sensu/rabbitmq_ssl_cert.pem",
      private_key_file: "/etc/sensu/rabbitmq_ssl_key.pem"
      }

and

foo::sensu::sentinels:
  - host: 192.168.113.9
    port: 26379
  - host: 192.168.113.10
    port: 26379
  - host: 192.168.113.11
    port: 26379

Don't use any rabbitmq_foo parameters except for rabbitmq_cluster. Manage the SSL certs for RabbitMQ outside the sensu module if you use SSL.

Hope that helps :)

ghoneycutt commented 7 years ago

Thanks @b-0-b for showing us the working config!

johanek commented 7 years ago

I too had this problem, and undefining any rabbitmq_foo parameters and manually managing the SSL certificates fixed it.

It's a bit of a shame that setting rabbitmq_ssl_cert_chain and rabbitmq_ssl_private_key to the content of the certificates doesn't write the file without disturbing the rabbitmq_cluster config, but such is life.

Thanks for figuring this out @b-0-b!

csoleimani commented 7 years ago

I've tried updating to the latest version of the module and am still facing this issue. Here is what my code looks like: Hiera:

sensu::rabbitmq_cluster:
  - host: 172.31.32.154
    port: 5672
    user: sensu
    password: "%{lookup('sensu::rabbitmq_password')}"
    vhost: 'sensu'
  - host: 172.31.33.158
    port: 5672
    user: sensu
    password: "%{lookup('sensu::rabbitmq_password')}"
    vhost: 'sensu'
  - host: 172.31.33.70
    port: 5672
    user: sensu
    password: "%{lookup('sensu::rabbitmq_password')}"
    vhost: 'sensu'

Manifest:

$rabbitmq_cluster = lookup('sensu::rabbitmq_cluster')

class { '::sensu':
  enterprise                   => true,
  enterprise_version           => $::sensu::enterprise_version,
  enterprise_dashboard         => true,
  enterprise_dashboard_version => $::sensu::enterprise_dashboard_version,
  enterprise_dashboard_host    => 'localhost',
  enterprise_dashboard_port    => '3000',
  enterprise_dashboard_refresh => '5',
  enterprise_dashboard_ldap    => {
  'basedn'              => $basedn,
  'groupbasedn'         => $groupbasedn,
  'userbasedn'          => $userbasedn,
  'binduser'            => $binduser,
  'bindpass'            => $bindpass,
  'server'              => $adserver,
  'port'                => $adport,
  'roles'               => [{
  'name'    => 'operators',
  'members' => ['operators'],
  },
  {
  'name'     => 'guests',
  'members'  => ['guests'],
  'readonly' => true,
  }
  ],
  'security'            => $security,
  'userattribute'       => $userattribute,
  'disablenestedgroups' => true,
  },
  enterprise_user              => 'enterprise-user',
  enterprise_pass              => 'enterprise-password',
  install_repo                 => true,
  log_level                    => $log_level,
  purge                        => {
    config => true,
  },
  rabbitmq_cluster             => $rabbitmq_cluster,
  redis_db                     => $redis_db,
  redis_host                   => $redis_host,
  version                      => $client_version,
}

Do you see any issues with my code? Could it be possible that I'm facing the issue because I'm using enterprise?

jeffmccune commented 7 years ago

I'm working on this and able to reproduce the error. Thank you so much @b-0-b for the wonderful bug report. I can't express how wonderful it is working on a project like this where the issues are so clearly reported, it's saved me tremendous time trying to understand and reproduce the issue. With this one, and other issues I've recently worked on, it's been downright fun to work them through. Thanks again!

Here's a partial backtrace leading to the issue in the provider, obtained with puppet apply --trace. This is triggered when specifying rabbitmq_password with rabbitmq_cluster as described:

Error: /Stage[main]/Sensu::Rabbitmq::Config/Sensu_rabbitmq_config[sensu-server.example.com]: Could not evaluate: no implicit conversion of String into Integer
/vagrant/lib/puppet/provider/sensu_rabbitmq_config/json.rb:137:in `[]'
/vagrant/lib/puppet/provider/sensu_rabbitmq_config/json.rb:137:in `password'

The provider behavior assumes conf['rabbitmq'] is a hash on line 137. The error occurs when conf['rabbitmq'] is an Array, which lines up with rabbitmq_cluster being an array in the reproduction case.

I'll pick back up tomorrow coming up with a game plan to improve this surprising behavior.

jeffmccune commented 7 years ago

The way the module is currently implemented, setting rabbitmq_ class parameters is mutually exclusive with setting rabbitmqcluster. #742 changes the behavior to avoid the error by ignoring the sensu::rabbitmq class parameters when rabbitmq_cluster is defined.

jeffmccune commented 7 years ago

PR #742 address the original issue reported by @b-0-b . @csoleimani I'm taking a look at the related issue you reported with sensu-enterprise now. Thanks for the example code.

jeffmccune commented 7 years ago

Hi @csoleimani

I've looked into the issue you're facing and you're running into the same issue originally reported. Consider this hiera data you have:

password: "%{lookup('sensu::rabbitmq_password')}"

This means sensu::rabbitmq_password exists, which will have the same effect as specifying the rabbitmq_password sensu class parameter. This, in turn, triggers the issue originally reported.

The change in #742 should fix the issue for you. The change causes sensu::rabbitmq_password to be ignored. With your code the password will take effect in the cluster configuration. Please keep an eye here for the released version, or give the code in #724 a shot and let us know if it's still broken for you.

Thanks for the clear information about how to reproduce your issue.

ghoneycutt commented 7 years ago

Fix implemented in v2.24.0