Error: JsonRpcConnection: Error while sending JSON-RPC message for identity

cde0 commented 2 years ago

Describe the bug

I start icinga but i get this Error: JsonRpcConnection: Error while sending JSON-RPC message for identity 'satellite2' Error: Broken pipe And it show that Server is Down.

Topology: master -> satellite -> monitor_server

[2022-01-03 14:34:59 +0000] information/ApiListener: Finished syncing endpoint 'satellite2' in zone 'satellite2'.
[2022-01-03 14:34:59 +0000] warning/JsonRpcConnection: Error while sending JSON-RPC message for identity 'satellite2'
Error: Broken pipe

Stacktrace:
 0# __cxa_throw in /usr/lib64/icinga2/sbin/icinga2
 1# boost::asio::detail::coro_async_result<boost::asio::executor_binder<void (*)(), boost::asio::executor>, unsigned long>::get() in /usr/lib64/icinga2/sbin/icinga2
 2# boost::asio::async_result<std::decay<boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >&>::type, void (boost::system::error_code, unsigned long)>::return_type boost::asio::buffered_write
_stream<icinga::UnbufferedAsioTlsStream>::async_flush<boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >&>(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::as
io::executor> >&) in /usr/lib64/icinga2/sbin/icinga2
 3# icinga::JsonRpcConnection::WriteOutgoingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib64/icinga2/sbin/icinga2
 4# 0x0000000000AFE697 in /usr/lib64/icinga2/sbin/icinga2
 5# 0x0000000000B06D89 in /usr/lib64/icinga2/sbin/icinga2
 6# make_fcontext in /lib64/libboost_context.so.1.69.0
[2022-01-03 14:34:59 +0000] warning/JsonRpcConnection: API client disconnected for identity 'satellite2'
[2022-01-03 14:34:59 +0000] warning/ApiListener: Removing API client for endpoint 'satellite2'. 0 API clients left.

To Reproduce

IDK :)

Expected behavior

Work, With out a crash

Screenshots

If applicable, add screenshots to help explain your problem.

Your Environment

Include as many relevant details about the environment you experienced the problem in

Version used (icinga2 --version): icinga2 - The Icinga 2 network monitoring daemon (version: 2.13.2-1)

Copyright (c) 2012-2022 Icinga GmbH (https://icinga.com/) License GPLv2+: GNU GPL version 2 or later https://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

System information: Platform: CentOS Linux Platform version: 7 (Core) Kernel: Linux Kernel version: 3.10.0-1160.25.1.el7.x86_64 Architecture: x86_64

Build information: Compiler: GNU 4.8.5 Build host: runner-hh8q3bz2-project-322-concurrent-0 OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017

Operating System and version: CentOS 7
Enabled features (icinga2 feature list): Enabled features: api checker mainlog
Icinga Web 2 version and modules (System - About):
Icinga Web 2 Version 2.9.5 Git commit 053971c99dc1a4510beb64a888ea695cc14032dc PHP Version 7.2.24-0ubuntu0.18.04.10 Git commit date 2021-11-18 Copyright © 2013-2022 Icinga GmbH
Config validation (icinga2 daemon -C): is return with out Error
If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

loosi commented 2 years ago

it seems to be an issue with icinga2 master > 2.13.0 in combination with older certain slaves/clients (older OS but up-to-date icinga2 version). I downgraded our master to 2.12.5 and now its working again. Nothing to hold on forever, but maybe someone else can figure out the issue (possibly something related to #8718 ?)

Al2Klimov commented 1 year ago

@cde0 Indeed, what are your Icinga 2 versions per node?

Wintermute2k6 commented 1 year ago

[ref/NC] #781426 Agent Version: v2.11.2 Master Version: v2.13.6-1

Al2Klimov commented 1 year ago

No 2.12 satellite in-between? That’s not supported!

Al2Klimov commented 9 months ago

it seems to be an issue with icinga2 master > 2.13.0 in combination with older certain slaves/clients (older OS but up-to-date icinga2 version).

Please be more specific on the versions.

the issue (possibly something related to #8718 ?)

Unlikely. Since v2.11, only TLSv1.2 is supported. That support hasn't been dropped by #8718.

Al2Klimov commented 5 months ago

I'm afraid such log messages indicate a remotely closed connection and only the respective peer's log at that time can say why it's been closed.

Icinga / icinga2