search

centreon / centreon-archived

Centreon is a network, system and application monitoring tool. Centreon is the only AIOps Platform Providing Holistic Visibility to Complex IT Workflows from Cloud to Edge.
https://www.centreon.com
GNU General Public License v2.0
575 stars 240 forks source link

Pollers not running after upgrading from 2.7.13 to 2.8.16 #5923

Closed fallais closed 6 years ago

fallais commented 6 years ago

Hello,

I upgraded from 2.7.13 to 2.8.16 by following this procedure : https://documentation.centreon.com/docs/centreon/en/2.8.x/upgrade/from_packages.html

Everything went fine, but my pollers are not showing up now. The version seems not to be updated. And it is in NO state, but if I check in CLI on the pollers, they are running.

centreon

I tried to search in all the issue, especially this one : https://github.com/centreon/centreon-engine/issues/86. But no luck.

OS is CentOS 6.9

Could you please help ? Thanks a lot.

lpinsivy commented 6 years ago

Do you have Centreon Broker temporary files on your pollers?

fallais commented 6 years ago

Thanks a lot for helping me. I'm really sorry but I do not know what is a Centreon Broker temporary files..

How can I check this please ?

fallais commented 6 years ago

Let me add that OS is CentOS 6.9

fallais commented 6 years ago

I restored my snapshot and tried to do it again, I followed exactly the procedure, I deleted all the retention files before doing it (I found them with the command find / -name *.retention).

But the result is the same. The Last Updated of the pollers is yellow. And the hosts and services are not updated anymore.

Please help, thanks.

Sims24 commented 6 years ago

Hello @fallais ,

Looks like your pollers still in old centreon-engine version and then also old centreon-broker version. You need to upgrade your poller to have Centreon-engine 1.8.x and Centreon-broker 3.0.X.

There is no backward compatibility between major version of scheduler and broker.

Hope this help

fallais commented 6 years ago

Hello @Sims24, thanks a lot for the answer. I did not say it but it followed the procedure for all my pollers as well..

Sims24 commented 6 years ago

Can you provide result of :

on pollers

grep cbmod /var/log/centreon-broker/centengine.log

on central

Send logfiles in /var/log/centreon-broker/ directory ?

fallais commented 6 years ago

Ok, here is what you asked.

Pollers

There is no /var/log/centreon-broker/centengine.log on pollers. Only v-srv-centreon-pooler-001-broker-master.log and watchdog.log

But there is :

[root@v-srv-centreon-pooler-001 yum.repos.d]# grep cbmod /var/log/centreon-engine/centengine.log
[1511957923] [1730] Event broker module '/usr/lib64/nagios/cbmod.so' deinitialized successfully
[1511958511] [7466] Event broker module '/usr/lib64/nagios/cbmod.so' initialized successfully

cat /var/log/centreon-broker/v-srv-centreon-pooler-001-broker-master.log

Centreon Broker 2.11.8 log file closed
Centreon Broker 3.0.11 log file opened
[1511958511] config:  state applier: broker_name is not  valid: allowed characters are ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 -_

Central

cat /var/log/centreon-broker/central-broker-master.log

...
...
Centreon Broker 2.11.8 log file closed
Centreon Broker 3.0.11 log file opened
[1511958505] config:  module applier: loading directory '/usr/share/centreon/lib/centreon-broker'

cat /var/log/centreon-broker/central-module-master.log

...
...
Centreon Broker 2.11.8 log file closed
Centreon Broker 3.0.11 log file opened
[1511958498] config:  log applier: applying 1 logging objects
[1511958498] config:  module applier: loading directory '/usr/share/centreon/lib/centreon-broker'
[1511958498] error:   TCP: could not connect to localhost:5669: Connection refused

cat /var/log/centreon-broker/central-rrd-master.log

...
...
Centreon Broker 2.11.8 log file closed
Centreon Broker 3.0.11 log file opened
[1511958505] config:  module applier: loading directory '/usr/share/centreon/lib/centreon-broker'

cat /var/log/centreon-broker/watchdog.log

Centreon Broker 3.0.11 log file opened
[1511958505] info:    watchdog: starting process 'central-broker-master'
[1511958505] info:    watchdog: process 'central-broker-master' started (PID 1238)
[1511958505] info:    watchdog: starting process 'central-rrd-master'
[1511958505] info:    watchdog: process 'central-rrd-master' started (PID 1239)

I can see that pollers are doing their job because I see new host statuses in /var/log/centreon-engine/centengine.log.

The proof that are the versions are the same :

[root@v-srv-centren-pooler-002 yum.repos.d]# rpm -qa | grep -i centreon
centreon-release-3.4-4.el6.noarch
centreon-broker-core-3.0.11-1.el6.x86_64
centreon-trap-2.8.16-8.el6.noarch
centreon-plugin-Network-Cisco-Standard-Snmp-20170808-1.el6.noarch
centreon-plugin-Applications-Databases-Mysql-20170807-1.el6.noarch
centreon-plugin-Applications-Protocol-Ftp-20170807-1.el6.noarch
centreon-clib-1.4.2-1.el6.x86_64
centreon-broker-storage-3.0.11-1.el6.x86_64
centreon-engine-daemon-1.8.1-1.el6.x86_64
centreon-engine-extcommands-1.8.1-1.el6.x86_64
centreon-broker-cbmod-3.0.11-1.el6.x86_64
centreon-plugin-Applications-Monitoring-Centreon-Poller-20170807-1.el6.noarch
centreon-perl-libs-2.8.16-8.el6.noarch
centreon-plugins-2.8.16-8.el6.noarch
centreon-plugin-Applications-Protocol-Dns-20170807-1.el6.noarch
centreon-plugin-Operatingsystems-Linux-Snmp-20170807-1.el6.noarch
centreon-connector-perl-1.1.3-1.el6.x86_64
centreon-plugin-Applications-Monitoring-Centreon-Map4-Jmx-20170807-1.el6.noarch
centreon-connector-1.1.3-1.el6.x86_64
centreon-plugin-Applications-Monitoring-Centreon-Database-20170807-1.el6.noarch
centreon-poller-centreon-engine-2.8.16-8.el6.noarch
centreon-broker-cbd-3.0.11-1.el6.x86_64
centreon-common-2.8.16-8.el6.noarch
centreon-broker-3.0.11-1.el6.x86_64
centreon-engine-1.8.1-1.el6.x86_64
centreon-plugin-Applications-Monitoring-Centreon-Central-20170807-1.el6.noarch
centreon-plugin-Hardware-Ups-Standard-Rfc1628-Snmp-20170807-1.el6.noarch
centreon-plugin-Operatingsystems-Windows-Snmp-20170807-1.el6.noarch
centreon-connector-ssh-1.1.3-1.el6.x86_64
centreon-plugin-Hardware-Printers-Generic-Snmp-20170807-1.el6.noarch
centreon-plugin-Applications-Protocol-Http-20170807-1.el6.noarch
centreon-plugin-Applications-Protocol-Ldap-20170807-1.el6.noarch
[root@v-srv-centreon-pooler-001 yum.repos.d]# rpm -qa | grep -i centreon
centreon-trap-2.8.16-8.el6.noarch
centreon-plugin-Hardware-Printers-Generic-Snmp-20170807-1.el6.noarch
centreon-plugin-Applications-Protocol-Ldap-20170807-1.el6.noarch
centreon-broker-3.0.11-1.el6.x86_64
centreon-broker-cbmod-3.0.11-1.el6.x86_64
centreon-plugin-Operatingsystems-Linux-Snmp-20170807-1.el6.noarch
centreon-broker-storage-3.0.11-1.el6.x86_64
centreon-plugins-2.8.16-8.el6.noarch
centreon-plugin-Operatingsystems-Windows-Snmp-20170807-1.el6.noarch
centreon-plugin-Applications-Protocol-Ftp-20170807-1.el6.noarch
centreon-broker-cbd-3.0.11-1.el6.x86_64
centreon-release-3.4-4.el6.noarch
centreon-broker-core-3.0.11-1.el6.x86_64
centreon-engine-extcommands-1.8.1-1.el6.x86_64
centreon-perl-libs-2.8.16-8.el6.noarch
centreon-plugin-Network-Cisco-Standard-Snmp-20170808-1.el6.noarch
centreon-plugin-Applications-Protocol-Dns-20170807-1.el6.noarch
centreon-plugin-Applications-Monitoring-Centreon-Map4-Jmx-20170807-1.el6.noarch
centreon-plugin-Applications-Databases-Mysql-20170807-1.el6.noarch
centreon-connector-1.1.3-1.el6.x86_64
centreon-common-2.8.16-8.el6.noarch
centreon-engine-1.8.1-1.el6.x86_64
centreon-plugin-Applications-Protocol-Http-20170807-1.el6.noarch
centreon-poller-centreon-engine-2.8.16-8.el6.noarch
centreon-clib-1.4.2-1.el6.x86_64
centreon-engine-daemon-1.8.1-1.el6.x86_64
centreon-plugin-Applications-Monitoring-Centreon-Poller-20170807-1.el6.noarch
centreon-plugin-Hardware-Ups-Standard-Rfc1628-Snmp-20170807-1.el6.noarch
centreon-plugin-Applications-Monitoring-Centreon-Central-20170807-1.el6.noarch
centreon-plugin-Applications-Monitoring-Centreon-Database-20170807-1.el6.noarch
centreon-connector-perl-1.1.3-1.el6.x86_64
centreon-connector-ssh-1.1.3-1.el6.x86_64
[root@v-srv-centreon-central yum.repos.d]# rpm -qa | grep -i centreon
centreon-plugin-Applications-Monitoring-Centreon-Database-20170807-1.el6.noarch
centreon-poller-centreon-engine-2.8.16-8.el6.noarch
centreon-widget-graph-monitoring-1.5.2-3.el6.noarch
centreon-clib-1.4.2-1.el6.x86_64
centreon-broker-core-3.0.11-1.el6.x86_64
centreon-engine-extcommands-1.8.1-1.el6.x86_64
centreon-connector-perl-1.1.3-1.el6.x86_64
centreon-connector-ssh-1.1.3-1.el6.x86_64
centreon-plugin-Applications-Protocol-Http-20170807-1.el6.noarch
centreon-license-manager-1.0.1-3.el6.noarch
centreon-widget-servicegroup-monitoring-1.6.0-1.el6.noarch
centreon-engine-daemon-1.8.1-1.el6.x86_64
centreon-broker-cbd-3.0.11-1.el6.x86_64
centreon-plugin-Network-Cisco-Standard-Snmp-20170808-1.el6.noarch
centreon-plugin-Applications-Databases-Mysql-20170807-1.el6.noarch
centreon-plugin-Operatingsystems-Windows-Snmp-20170807-1.el6.noarch
centreon-plugin-Applications-Monitoring-Centreon-Map4-Jmx-20170807-1.el6.noarch
centreon-plugin-meta-2.8.16-8.el6.noarch
centreon-2.8.16-8.el6.noarch
centreon-widget-service-monitoring-1.6.1-2.el6.noarch
centreon-release-3.4-4.el6.noarch
centreon-broker-3.0.11-1.el6.x86_64
centreon-perl-libs-2.8.16-8.el6.noarch
centreon-broker-cbmod-3.0.11-1.el6.x86_64
centreon-plugin-Applications-Monitoring-Centreon-Central-20170807-1.el6.noarch
centreon-plugin-Hardware-Ups-Standard-Rfc1628-Snmp-20170807-1.el6.noarch
centreon-plugin-Operatingsystems-Linux-Snmp-20170807-1.el6.noarch
centreon-plugin-Applications-Protocol-Ftp-20170807-1.el6.noarch
centreon-web-2.8.16-8.el6.noarch
centreon-base-config-centreon-engine-2.8.16-8.el6.noarch
centreon-widget-hostgroup-monitoring-1.6.0-1.el6.noarch
centreon-common-2.8.16-8.el6.noarch
centreon-broker-storage-3.0.11-1.el6.x86_64
centreon-trap-2.8.16-8.el6.noarch
centreon-engine-1.8.1-1.el6.x86_64
centreon-connector-1.1.3-1.el6.x86_64
centreon-pp-manager-2.2.0-4.el6.noarch
centreon-plugins-2.8.16-8.el6.noarch
centreon-plugin-Applications-Protocol-Dns-20170807-1.el6.noarch
centreon-plugin-Applications-Monitoring-Centreon-Poller-20170807-1.el6.noarch
centreon-plugin-Hardware-Printers-Generic-Snmp-20170807-1.el6.noarch
centreon-plugin-Applications-Protocol-Ldap-20170807-1.el6.noarch
centreon-widget-host-monitoring-1.6.1-1.el6.noarch
Sims24 commented 6 years ago

Your broker name is invalid, please post your xml config file from pooler-001 /etc/centreon-broker/.xml.

[1511958511] config: state applier: broker_name is not valid: allowed characters are ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 -_

Thanks a lot, Simon

fallais commented 6 years ago

Hello @Sims24 ,

Here it is. I hide some values with yyy.

[root@v-srv-centreon-pooler-001 ~]# cat /etc/centreon-broker/v-srv-centreon-pooler-001.yyy.yyy.fr-module.xml
<centreonBroker>
 <broker_id><![CDATA[4]]></broker_id>
 <broker_name><![CDATA[v-srv-centreon-pooler-001.yyy.yyy.fr-module]]></broker_name>
 <poller_id><![CDATA[2]]></poller_id>
 <poller_name><![CDATA[v-srv-centreon-pooler-001.yyy.yyy.fr]]></poller_name>
 <module_directory><![CDATA[/usr/share/centreon/lib/centreon-broker]]></module_directory>
 <log_timestamp><![CDATA[1]]></log_timestamp>
 <log_thread_id><![CDATA[0]]></log_thread_id>
 <event_queue_max_size><![CDATA[100000]]></event_queue_max_size>
 <command_file><![CDATA[]]></command_file>
 <cache_directory><![CDATA[/var/lib/centreon-broker]]></cache_directory>
 <logger>
  <name><![CDATA[/var/log/centreon-broker/v-srv-centreon-pooler-001-broker-master.log]]></name>
  <config><![CDATA[yes]]></config>
  <debug><![CDATA[no]]></debug>
  <error><![CDATA[yes]]></error>
  <info><![CDATA[no]]></info>
  <level><![CDATA[low]]></level>
  <max_size><![CDATA[50000000000]]></max_size>
  <type><![CDATA[file]]></type>
 </logger>
 <output>
  <name><![CDATA[v-srv-centreon-pooler-001.yyy.yyy.fr-module-master]]></name>
  <port><![CDATA[5669]]></port>
  <host><![CDATA[v-srv-centreon-central.yyy.yyy.fr]]></host>
  <retry_interval><![CDATA[60]]></retry_interval>
  <buffering_timeout><![CDATA[0]]></buffering_timeout>
  <protocol><![CDATA[bbdo]]></protocol>
  <tls><![CDATA[auto]]></tls>
  <negociation><![CDATA[yes]]></negociation>
  <one_peer_retention_mode><![CDATA[no]]></one_peer_retention_mode>
  <compression><![CDATA[auto]]></compression>
  <type><![CDATA[ipv4]]></type>
 </output>
 <stats>
  <type><![CDATA[stats]]></type>
  <name><![CDATA[v-srv-centreon-pooler-001.yyy.yyy.fr-module-stats]]></name>
  <json_fifo><![CDATA[/var/lib/centreon-broker/v-srv-centreon-pooler-001.yyy.yyy.fr-module-stats.json]]></json_fifo>
 </stats>
</centreonBroker>
[root@v-srv-centreon-pooler-001 ~]# cat /etc/centreon-broker/watchdog.xml
<centreonbroker>
 <log><![CDATA[/var/log/centreon-broker/watchdog.log]]></log>
</centreonbroker>
[root@v-srv-centreon-pooler-001 ~]# cat /etc/centreon-broker/poller-module.xml
<?xml version="1.0" encoding="UTF-8"?>
<conf>

 <!-- Global configuration options. -->
 <broker_id>XXX</broker_id>
 <broker_name>XXX</broker_name>
 <poller_id>XXX</poller_id>
 <poller_name>XXX</poller_name>
 <module_directory>/usr/share/centreon/lib/centreon-broker</module_directory>
 <event_queue_max_size>10000</event_queue_max_size>
 <flush_logs>no</flush_logs>
 <log_thread_id>no</log_thread_id>
 <log_timestamp>yes</log_timestamp>
 <cache_directory>/var/lib/centreon-broker</cache_directory>
 <command_file>/var/lib/centreon-broker/poller-module.cmd</command_file>

 <!-- Send monitoring information to central-broker.
      Get external commands and configuration files from central-broker. -->
 <output>
  <retry_interval>60</retry_interval>
  <type>tcp</type>
  <name>poller-module-output</name>
  <port>5668</port>
  <host>localhost</host>
  <protocol>bbdo</protocol>
  <filters>
   <category>neb</category>
   <category>extcmd:command_result</category>
  </filters>
 </output>

 <!-- Write Engine external commands issued by central-broker. -->
 <output>
  <retry_interval>60</retry_interval>
  <type>dumper</type>
  <tagname>extcommands-engine-$POLLERID$</tagname>
  <path>/var/lib/centreon-engine/rw/centengine.cmd</path>
  <name>poller-module-extcommands-engine</name>
 </output>

 <!-- Write Broker configuration files issued by central-broker. -->
 <output>
  <retry_interval>60</retry_interval>
  <type>dumper</type>
  <tagname>cfg-broker-$POLLERID$</tagname>
  <path>/etc/centreon-broker/$FILENAME$</path>
  <name>poller-module-cfg-broker</name>
 </output>

 <!-- Write Engine configuration files issued by central-broker. -->
 <output>
  <retry_interval>60</retry_interval>
  <type>dumper</type>
  <tagname>cfg-engine-$POLLERID$</tagname>
  <path>/etc/centreon-engine/$FILENAME$</path>
  <name>poller-module-cfg-engine</name>
 </output>

 <!-- Log file. -->
 <logger>
  <config>yes</config>
  <debug>no</debug>
  <error>yes</error>
  <info>yes</info>
  <level>low</level>
  <type>file</type>
  <name>/var/log/centreon-broker/poller-module.log</name>
 </logger>

 <!-- Statistics. -->
 <stats>
  <type>stats</type>
  <name>poller-module-stats</name>
  <fifo>/var/lib/centreon-broker/poller-module.stats</fifo>
 </stats>

</conf>
[root@v-srv-centreon-pooler-001 ~]# cat /etc/centreon-broker/central-rrd.xml
<?xml version="1.0" encoding="UTF-8"?>
<conf>

 <!-- Global configuration options. -->
 <broker_id>2</broker_id>
 <broker_name>central-rrd</broker_name>
 <poller_id>1</poller_id>
 <poller_name>Central</poller_name>
 <module_directory>/usr/share/centreon/lib/centreon-broker</module_directory>
 <event_queue_max_size>10000</event_queue_max_size>
 <flush_logs>no</flush_logs>
 <log_thread_id>no</log_thread_id>
 <log_timestamp>yes</log_timestamp>
 <cache_directory>/var/lib/centreon-broker</cache_directory>
 <command_file>/var/lib/centreon-broker/central-rrd.cmd</command_file>

 <!-- Read performance data. -->
 <input>
  <retry_interval>60</retry_interval>
  <type>tcp</type>
  <name>central-rrd-input</name>
  <port>5670</port>
  <protocol>bbdo</protocol>
  <tls>auto</tls>
  <negociation>yes</negociation>
  <compression>auto</compression>
 </input>

 <!-- Write RRD files. -->
 <output>
  <retry_interval>60</retry_interval>
  <metrics_path>/var/lib/centreon/metrics</metrics_path>
  <status_path>/var/lib/centreon/status</status_path>
  <path></path>
  <port></port>
  <write_metrics>yes</write_metrics>
  <write_status>yes</write_status>
  <type>rrd</type>
  <name>central-rrd-files</name>
 </output>

 <!-- Log file. -->
 <logger>
  <config>yes</config>
  <debug>no</debug>
  <error>yes</error>
  <info>yes</info>
  <level>low</level>
  <type>file</type>
  <name>/var/log/centreon-broker/central-rrd.log</name>
 </logger>

 <!-- Statistics. -->
 <stats>
  <type>stats</type>
  <name>central-rrd-stats</name>
  <fifo>/var/lib/centreon-broker/central-rrd.stats</fifo>
 </stats>

</conf>
fallais commented 6 years ago

Seeing this I now understand the dots are not allowed.. right ? How can I fix this please ?

fallais commented 6 years ago

If I change the broker name in the GUI and generate the configuration again, I do not have this error anymore but a new one.

Centreon Broker 3.0.11 log file opened [1512049460] config: state applier: endpoint name '.' is not valid: allowed characters are ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789 -_

What is the endpoint name please ? How can I change it ?

fallais commented 6 years ago

I fixed it. I needed to change both the Name in the General tab of the broker configuration and the Name in the output tab of the broker configuration.

Then, I generated the configuration and exported it. Finally, do not forget a service centengine restart and service cbd restart

Thanks @Sims24 for your help.