search

mocdaniel / dashing-icinga2

Dashing dashboard for Icinga 2 using the REST API
MIT License
204 stars 47 forks source link

icinga v2.10.3 - Unable to process available data, they're already being processed in another thread #72

Closed sharkyzz closed 5 years ago

sharkyzz commented 5 years ago

Since I have upgraded Icinga from v2.10.2 to v2.10.3 i'm experiencing issues with Dashing in combination with Icinga. Which causing the warnings below and results in critical warnings 'too many open files' in Icinga eventually.

[2019-03-02 14:52:45 +0100] critical/checker: Exception occurred while checking 'xxx': Error: Function call 'pipe2' failed with error code 24, 'Too many open files'
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Request: GET /v1/status/CIB (from [127.0.0.1]:54240), user: dashing)
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: HTTP client disconnected (from [127.0.0.1]:54240)
[2019-03-02 20:27:25 +0100] information/ApiListener: New client connection from [127.0.0.1]:54244 (no client certificate)
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts (from [127.0.0.1]:54244), user: dashing)
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
[2019-03-02 20:27:25 +0100] information/HttpServerConnection: Unable to process available data, they're already being processed in another thread
dnsmichi commented 5 years ago

That's a problem in Icinga 2 then, the dashing jobs just connect and pull the data in a defined interval in jobs/icinga2.rb. You could try raising the interval there.

aledr commented 5 years ago

Almost same problem here on 2.10.4 but worst. It causes segmentation fault on Icinga2 (icinga2_ido killed). @sharkyzz did you find the cause?

sharkyzz commented 5 years ago

@aledr in my case the problem was related to v2.10.3. After I've upgraded to v2.10.4 the problem was solved. Can you be more specific about your problem, maybe I can help ?

dnsmichi commented 5 years ago

Please move this discussion over to https://community.icinga.com :)

sharkyzz commented 5 years ago

Yes, it already didn't feel right proceeding here :)