Closed StefThomas closed 2 years ago
More information : In fact, while Broker is at the same version on both plateformes, Centreon Web is not. I have a 20.04.12 on the plateforme which works, and 20.04.13 on the plateforme which doesn’t.
I’ll try to downgrade, if I can.
Downgrade didn’t fix the issue. Also, I upgraded to centreon-web 20.04.13 on the working plateforme and I can see no bug on this one.
The errros messages I have are in centreon-broker-master.log and centreon-rrd-master.log and are the following (they repeat a lot, I just pasted a few here) :
[1620979447] error: SQL: service group 73 does not exist - insertion before insertion of members
[1620979448] error: mysql_connection: could not store host group membership: Duplicate entry '5240-243' for key 'host_id'
[1620979518] error: RRD: ignored update error in file '/var/lib/centreon/status/5879818.rrd': /var/lib/centreon/status/5879818.rrd: illegal attempt to update using time 1620825350 when l
ast update time is 1620825350 (minimum one second step)
Any idea?
After some other tests I can tell it is updating until the following error message appears in /var/log/centreon-broker/central-broker-master.log :
[1620991591] error: mysql_connection: Duplicate entry '333148' for key 'PRIMARY'
[1620991591] error: conflict_manager: error in the main loop: storage: insertion of metric 'count' of index 39066258 failed: Duplicate entry '333148' for key 'PRIMARY'
Then nothing more is written to this file. The cbd and cbdw are still running :
● cbd.service - Centreon Broker watchdog
Loaded: loaded (/etc/systemd/system/cbd.service; enabled; vendor preset: disabled)
Active: active (running) since ven. 2021-05-14 13:26:01 CEST; 23min ago
Process: 22318 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 32494 (cbwd)
CGroup: /system.slice/cbd.service
├─32494 /usr/sbin/cbwd /etc/centreon-broker/watchdog.json
├─32495 /usr/sbin/cbd /etc/centreon-broker/central-broker.json
└─32496 /usr/sbin/cbd /etc/centreon-broker/central-rrd.json
mai 14 13:26:01 SR082501CTI3700.hm.dm.ad systemd[1]: Started Centreon Broker watchdog.
mai 14 13:26:01 SR082501CTI3700.hm.dm.ad cbwd[32494]: [1620991561] info: file: module for Centreon Broker 20.04.13
mai 14 13:26:01 SR082501CTI3700.hm.dm.ad cbwd[32494]: [1620991561] config: log applier: applying 1 logging objects
mai 14 13:26:01 SR082501CTI3700.hm.dm.ad cbwd[32494]: [1620991561] info: file: module for Centreon Broker 20.04.13
mai 14 13:26:01 SR082501CTI3700.hm.dm.ad cbwd[32494]: [1620991561] config: log applier: applying 1 logging objects
I have this error systematically when I restart the cbd service.
I have manually forced some checks, then I can see it has updated some data, until the error above happened :
@StefThomas I sent you a Webex link through the support portal. Please join it so we can troubleshoot this together.
Simon
We have same issue on latest 20.10.5 Broker and Centreon Web 20.10.8. The problem occurs on our system when you add a Host to a Hostgroup. This will cause the problem on Central Broker and blocks all other Events. Is there any hotfix available?
This will only happen on hostgroup created after the update.
If have done some test, broker has some problems to update/insert the table hosts_hostgroups:
Have done some other tests, problem is also present on latest Centreon 21.04:
[2021-07-02 08:29:37.850] [core] [debug] accept ('central-broker-master-input') failed.
[2021-07-02 08:29:40.850] [core] [debug] accept ('central-broker-master-input') failed.
[2021-07-02 08:29:43.088] [sql] [error] mysql_connection: could not store host group membership: Lock wait timeout exceeded; try restarting transaction
[2021-07-02 08:29:43.851] [core] [debug] accept ('central-broker-master-input') failed.
[2021-07-02 08:29:46.851] [core] [debug] accept ('central-broker-master-input') failed.
Problem is not related only when you assign hosts to a new created hostgroup but it will happen also with existing hostgroups, but not any time, seems sporadic. But when will happen, broker is blocked
On this master-broker debug log you can see the issue starting from [2021-07-02 08:39:35]
Any news about this problem? Also with latest updates
Hi all, We have released quite a lot of fixes regarding broker connection stability issues (see release notes) recently. Do you still face such errors after updating to the latest 20.10/21.04 (the fixes are going to be released in 20.04 as well quite soon)? Regards,
Hi,
I’ve updated a latest 19.04 plateforme to 20.04.13 (following this documentation: https://docs.centreon.com/20.04/fr/upgrade/upgrade-from-19-04.html). While I had no problem doing so on another plateforme, on this one Centreon isn’t working anymore. I use ZMQ as the Gogrgone communication protocole. There don’t seem to have any problem with Gorgone. If I order a restart from the UI (central), the engines are restarted, even on the distant poller, but the central can’t see any update :
And gorgoned’s log shows no error. I have no error in /var/opt/rh/rh-php72/log/php-fpm/centreon-error.log too.
I disabled the remote pollers while I’m trying to get rid on the problem.
I also removed the cache as advised to do so by the support :
All the supervision data are stacking in /var/lib/centreon-broker/ :
(I already moved some of those files on a backup space so the filesystem would not fill during the yesterday holiday (hence the index not starting from 1).
I activated full debug on the broker. This is the only error message I have, in /var/log/centreon-broker/central-broker-master.log :
And the suffixe is increasing if I order another restart :
In the central-module-master.log I have this message :
But the file exists :
I also had this message :
But it seems not to be related. This is what the support told me and I can’t see it anymore today. It was probably caused by some of the data I moved to gain space on /var/lib/centreon-broker/
I’ll continue to try to troubleshoot this issue, check again once more the settings of the broker, but I don’t have any idea anymore. I already checked the configuration multiple times. I have no clue.
The other plateforme had been upgrade from the latest 19.04 to Centreon 20.04.12 (it was some weeks ago), but it has been updated to those same versions (20.04.13) and I experience no problem on this one.