icinga-migration
commented
8 years ago Updated by mfriedrich on 2015-11-25 14:25:25 +00:00
- Category set to API
Closed icinga-migration closed 8 years ago
icinga-migration
commented
8 years ago Updated by mfriedrich on 2015-11-25 14:25:25 +00:00
icinga-migration
commented
8 years ago Updated by jonan on 2015-11-28 08:13:56 +00:00
I can also confirm resource exhaustion using the API when running the following Icinga environment:
Icinga version: v2.4.1-42-g6a83703 on Ubuntu 14.04
Hitting any api call seems to do it. You can reproduce as follows:
Monitor the Icinga thread count with:
$ watch -n1 'ps -o nlwp 'Query API using Apache bench:
$ ab -A root:icinga -n 17000 -c 100 https://localhost:5665/v1Icinga dies after hitting the max processes limit of 16384.
Logs show the following errors when failure begins:
> warning/HttpServerConnection: Error while reading Http request: Error: boost::thread_resource_error: Resource temporarily unavailable
> critical/checker: Exception occured while checking 'host-22!ssh': Error: Function call 'fork' failed with error code 11, 'Resource temporarily unavailable'
icinga-migration
commented
8 years ago Updated by mfriedrich on 2015-11-30 16:59:20 +00:00
I'm able to reproduce the issue, though I need to look into it in deep when other projects are finished.
mbmif /usr/local/icinga2 (master) # lldb -- /usr/local/icinga2/lib/icinga2/sbin/icinga2 daemon
(lldb) target create "/usr/local/icinga2/lib/icinga2/sbin/icinga2"
Current executable set to '/usr/local/icinga2/lib/icinga2/sbin/icinga2' (x86_64).
(lldb) settings set -- target.run-args "daemon"
(lldb) r
Process 68017 launched: '/usr/local/icinga2/lib/icinga2/sbin/icinga2' (x86_64)
[2015-11-30 12:32:50 +0100] information/cli: Icinga application loader (version: v2.4.1-47-gab9218e)
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/icinga2.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/constants.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/zones.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/itl
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/command.conf
[2015-11-30 12:32:50 +0100] information/Utility: Loading library 'libmethods.dylib'
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/command-icinga.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/timeperiod.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/command-plugins.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/databases.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/hardware.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/ipmi.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/network-components.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/operating-system.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/virtualization.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/vmware.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/share/icinga2/include/plugins-contrib.d/web.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/api.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/checker.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/command.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/compatlog.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/graphite.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/ido-mysql.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/livestatus.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/mainlog.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/notification.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/features-enabled/perfdata.conf
[2015-11-30 12:32:50 +0100] information/Utility: Loading library 'libchecker.dylib'
[2015-11-30 12:32:50 +0100] information/Utility: Loading library 'libcompat.dylib'
[2015-11-30 12:32:50 +0100] information/Utility: Loading library 'libcompat.dylib'
[2015-11-30 12:32:50 +0100] information/Utility: Loading library 'libperfdata.dylib'
[2015-11-30 12:32:50 +0100] information/Utility: Loading library 'libdb_ido_mysql.dylib'
[2015-11-30 12:32:50 +0100] information/Utility: Loading library 'liblivestatus.dylib'
[2015-11-30 12:32:50 +0100] information/Utility: Loading library 'libnotification.dylib'
[2015-11-30 12:32:50 +0100] information/Utility: Loading library 'libperfdata.dylib'
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/api-users.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/app.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/aws.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/commands.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/comments.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/downtimes.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/groups.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/hosts.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/notifications.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/satellite.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/services.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/templates.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/test.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/timeperiods.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/conf.d/users.conf
[2015-11-30 12:32:50 +0100] information/config: []
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/10237.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/8183.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/dep.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/groups.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/mw.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/out.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/etc/icinga2/tests/random.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/include.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/include.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/../active.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/comments/mbmif.int.netways.de!bla.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448439527-0.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448445272-0.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/mbmif.int.netways.de!load!mbmif.int.netways.de-1448872589-2.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/bumsti1.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/google.com.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/google.com1.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/google.com2.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/google.com3.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/google.com4.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/i-42866686.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/i-43866687.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/hosts/michiathome1.conf
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/services/bumsti1!ping7.conf
[2015-11-30 12:32:50 +0100] information/ConfigItem: Committing config items
[2015-11-30 12:32:50 +0100] information/ApiListener: My API identity: mbmif.int.netways.de
[2015-11-30 12:32:50 +0100] warning/ConfigObject: Ignoring config object 'mbmif.int.netways.de-1448445272-0' of type 'Downtime' due to errors: Error: Validation failed for object 'master.dm.netways.de!load!mbmif.int.netways.de-1448445272-0' of type 'Downtime'; Attribute 'host_name': Object 'master.dm.netways.de' of type 'Host' does not exist.
Location: in /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448445272-0.conf: 1:0-1:66
/usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448445272-0.conf(1): object Downtime "mbmif.int.netways.de-1448445272-0" ignore_on_error {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448445272-0.conf(2): author = "icingaadmin"
/usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448445272-0.conf(3): comment = "Scheduled downtime for backup"
[2015-11-30 12:32:50 +0100] warning/ConfigObject: Ignoring config object 'mbmif.int.netways.de-1448439527-0' of type 'Downtime' due to errors: Error: Validation failed for object 'master.dm.netways.de!load!mbmif.int.netways.de-1448439527-0' of type 'Downtime'; Attribute 'host_name': Object 'master.dm.netways.de' of type 'Host' does not exist.
Location: in /usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448439527-0.conf: 1:0-1:66
/usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448439527-0.conf(1): object Downtime "mbmif.int.netways.de-1448439527-0" ignore_on_error {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448439527-0.conf(2): author = "icingaadmin"
/usr/local/icinga2/var/lib/icinga2/api/packages/_api/mbmif.int.netways.de-1442309540-1/conf.d/downtimes/master.dm.netways.de!load!mbmif.int.netways.de-1448439527-0.conf(3): comment = "Scheduled downtime for backup"
[2015-11-30 12:32:50 +0100] warning/ConfigObject: Ignoring config object 'foo' of type 'Comment' due to errors: Error: Comment 'mbmif.int.netways.de!abc!foo' references a host/service which doesn't exist.
Location: in /usr/local/icinga2/etc/icinga2/conf.d/comments.conf: 1:0-1:35
/usr/local/icinga2/etc/icinga2/conf.d/comments.conf(1): object Comment "foo" ignore_on_error {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/local/icinga2/etc/icinga2/conf.d/comments.conf(2): host_name = NodeName
/usr/local/icinga2/etc/icinga2/conf.d/comments.conf(3): service_name = "abc"
[2015-11-30 12:32:50 +0100] warning/ApplyRule: Apply rule 'satellite-host' (in /usr/local/icinga2/etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere!
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 Endpoint.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 Zone.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 ApiUser.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 70 Dependencies.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 Downtime.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 44 Services.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 3 ServiceGroups.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 3 TimePeriods.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 14 Users.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 3 UserGroups.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 26 Hosts.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 6 HostGroups.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 2 NotificationCommands.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 11 Notifications.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 154 CheckCommands.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 Comment.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 CompatLogger.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 PerfdataWriter.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 LivestatusListener.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2015-11-30 12:32:50 +0100] information/ScriptGlobal: Dumping variables to file '/usr/local/icinga2/var/cache/icinga2/icinga2.vars'
[2015-11-30 12:32:50 +0100] information/ConfigObject: Restoring program state from file '/usr/local/icinga2/var/lib/icinga2/icinga2.state'
[2015-11-30 12:32:50 +0100] information/ConfigObject: Restored 353 objects. Loaded 0 new objects without state.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Triggering Start signal for config items
[2015-11-30 12:32:50 +0100] information/ApiListener: Adding new listener on port '5665'
[2015-11-30 12:32:50 +0100] information/DbConnection: Resuming IDO connection: ido-mysql
[2015-11-30 12:32:50 +0100] information/LivestatusListener: Created UNIX socket in '/usr/local/icinga2/var/run/icinga2/cmd/livestatus'.
[2015-11-30 12:32:50 +0100] information/ConfigItem: Activated all objects.
[2015-11-30 12:32:50 +0100] information/ConfigCompiler: Compiling config file: /usr/local/icinga2/var/lib/icinga2/modified-attributes.conf
[2015-11-30 12:32:50 +0100] information/ApiListener: New client connection (no client certificate)
[2015-11-30 12:32:50 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts?host[]=mbmif.int.netways.de (root)
[2015-11-30 12:32:50 +0100] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.0')
[2015-11-30 12:32:50 +0100] information/ApiListener: New client connection (no client certificate)
[2015-11-30 12:32:50 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts?host[]=mbmif.int.netways.de (root)
...
[2015-11-30 12:34:22 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts?host[]=mbmif.int.netways.de (root)
[2015-11-30 12:34:22 +0100] information/ApiListener: New client connection (no client certificate)
[2015-11-30 12:34:22 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts?host[]=mbmif.int.netways.de (root)
[2015-11-30 12:34:22 +0100] information/ApiListener: New client connection (no client certificate)
[2015-11-30 12:34:22 +0100] warning/HttpServerConnection: Error while reading Http request: Error: boost::thread_resource_error: Resource temporarily unavailable
(0) 2 libbase.dylib 0x000000010086391a _ZN5boost6thread12start_threadEv + 122
(1) 3 libbase.dylib 0x0000000100860634 _ZN5boost12thread_group13create_threadINS_3_bi6bind_tIvNS_4_mfi3mf0IvN6icinga9WorkQueueEEENS2_5list1INS2_5valueIPS7_EEEEEEEEPNS_6threadET_ + 196
(2) 4 libbase.dylib 0x0000000100808cd2 _ZN6icinga9WorkQueue7EnqueueERKN5boost8functionIFvvEEEb + 194
(3) 5 libremote.dylib 0x0000000100440c18 _ZN6icinga20HttpServerConnection14ProcessMessageEv + 360
(4) 6 libremote.dylib 0x00000001004405b8 _ZN6icinga20HttpServerConnection20DataAvailableHandlerEv + 56
(5) 7 libremote.dylib 0x00000001003f91ed _ZN6icinga20HttpServerConnection5StartEv + 413
(6) 8 libremote.dylib 0x00000001003f6c9f _ZN6icinga11ApiListener24NewClientHandlerInternalERKN5boost13intrusive_ptrINS_6SocketEEERKNS_6StringENS_14ConnectionRoleE + 1647
(7) 9 libremote.dylib 0x00000001003f54d9 _ZN6icinga11ApiListener16NewClientHandlerERKN5boost13intrusive_ptrINS_6SocketEEERKNS_6StringENS_14ConnectionRoleE + 25
(8) 10 libboost_thread-mt.dylib 0x0000000100043715 _ZN5boost12_GLOBAL__N_112thread_proxyEPv + 53
(9) 11 libsystem_pthread.dylib 0x00007fff86b599b1 _pthread_body + 131
(10) 12 libsystem_pthread.dylib 0x00007fff86b5992e _pthread_body + 0
(11) 13 libsystem_pthread.dylib 0x00007fff86b57385 thread_start + 13
(0) Handling new API client connection
Context:
(0) Handling new API client connection
Process 68017 stopped
* thread #14: tid = 0x25ca81, 0x0000000100441f52 libremote.dylib`icinga::HttpRequest::Parse(this=0x0000000106915668, src=0x0000000106915ae0, may_wait=false) + 898 at httprequest.cpp:142, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
frame #0: 0x0000000100441f52 libremote.dylib`icinga::HttpRequest::Parse(this=0x0000000106915668, src=0x0000000106915ae0, may_wait=false) + 898 at httprequest.cpp:142
139 return false;
140 }
141
-> 142 m_Body->Write(src.Buffer, length_indicator);
143 src.DropData(length_indicator);
144 Complete = true;
145 return true;
(lldb)
thread #2074: tid = 0x25faba, 0x00007fff8ac86f5e libsystem_kernel.dylib`__psynch_cvwait + 10
* thread #2075: tid = 0x25fabd, 0x00007fff8ac86f5e libsystem_kernel.dylib`__psynch_cvwait + 10
(lldb) thr sel 2074
* thread #2074: tid = 0x25faba, 0x00007fff8ac86f5e libsystem_kernel.dylib`__psynch_cvwait + 10
frame #0: 0x00007fff8ac86f5e libsystem_kernel.dylib`__psynch_cvwait + 10
libsystem_kernel.dylib`__psynch_cvwait:
-> 0x7fff8ac86f5e <+10>: jae 0x7fff8ac86f68 ; <+20>
0x7fff8ac86f60 <+12>: movq %rax, %rdi
0x7fff8ac86f63 <+15>: jmp 0x7fff8ac823ef ; cerror_nocancel
0x7fff8ac86f68 <+20>: retq
(lldb) up
frame #1: 0x00007fff86b5a73d libsystem_pthread.dylib`_pthread_cond_wait + 767
libsystem_pthread.dylib`_pthread_cond_wait:
0x7fff86b5a73d <+767>: movl %eax, %ebx
0x7fff86b5a73f <+769>: movq %gs:0x0, %rdi
0x7fff86b5a748 <+778>: movl %r12d, %esi
0x7fff86b5a74b <+781>: callq 0x7fff86b5a3d5 ; _pthread_testcancel
(lldb)
frame #2: 0x000000010085efeb libbase.dylib`boost::condition_variable::wait(this=0x000000010346cf58, m=0x00007000416f7ce8) + 59 at condition_variable.hpp:73
70 detail::interruption_checker check_for_interruption(&internal_mutex,&cond);
71 guard.activate(m);
72 do {
-> 73 res = pthread_cond_wait(&cond,&internal_mutex);
74 } while (res == EINTR);
75 #else
76 //boost::pthread::pthread_mutex_scoped_lock check_for_interruption(&internal_mutex);
(lldb)
frame #3: 0x0000000100857fe7 libbase.dylib`icinga::WorkQueue::WorkerThreadProc(this=0x000000010346cf08) + 807 at workqueue.cpp:195
192
193 for (;;) {
194 while (m_Tasks.empty() && !m_Stopped)
-> 195 m_CVEmpty.wait(lock);
196
197 if (m_Stopped)
198 break;
(lldb) p *this
(icinga::WorkQueue) $4 = {
m_ID = 2035
m_ThreadCount = 1
m_Spawned = true
m_Mutex = {
m = (__sig = 1297437786, __opaque = "")
}
m_CVEmpty = {
internal_mutex = (__sig = 1297437786, __opaque = "")
cond = (__sig = 1129270852, __opaque = "")
}
m_CVFull = {
internal_mutex = (__sig = 1297437786, __opaque = "")
cond = (__sig = 1129270852, __opaque = "")
}
m_CVStarved = {
internal_mutex = (__sig = 1297437786, __opaque = "")
cond = (__sig = 1129270852, __opaque = "")
}
m_Threads = {
threads = size=1 {
[0] = 0x0000000105e20390
}
m = {
state = {
shared_count = 0
exclusive = false
upgrade = false
exclusive_waiting_blocked = false
}
state_change = {
m = (__sig = 1297437786, __opaque = "")
}
shared_cond = {
internal_mutex = (__sig = 1297437786, __opaque = "")
cond = (__sig = 1129270852, __opaque = "")
}
exclusive_cond = {
internal_mutex = (__sig = 1297437786, __opaque = "")
cond = (__sig = 1129270852, __opaque = "")
}
upgrade_cond = {
internal_mutex = (__sig = 1297437786, __opaque = "")
cond = (__sig = 1129270852, __opaque = "")
}
}
}
m_MaxItems = 0
m_Stopped = false
m_Processing = 0
m_Tasks = size=0 {}
m_ExceptionCallback = {
boost::function1 = {
boost::function_base = {
vtable = 0x0000000000000000
functor = {
obj_ptr = 0x0000000000000000
type = {
type = 0x0000000000000000
const_qualified = false
volatile_qualified = false
}
func_ptr = 0x0000000000000000
bound_memfunc_ptr = (memfunc_ptr = 0x0000000000000000, obj_ptr = 0x0000000000000000)
obj_ref = (obj_ptr = 0x0000000000000000, is_const_qualified = false, is_volatile_qualified = false)
data = '\0'
}
}
}
}
m_Exceptions = size=0 {}
m_StatusTimer = {
px = 0x0000000105b7abb0
}
}Updated by mfriedrich on 2015-12-15 11:10:54 +00:00
icinga-migration
commented
8 years ago Updated by mfriedrich on 2015-12-18 11:22:03 +00:00
icinga-migration
commented
8 years ago Updated by peikk0 on 2016-01-07 12:14:46 +00:00
I have the same issue on FreeBSD 10.2 with version 2.4.1
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-01-11 13:05:58 +00:00
We should fix that, though I currently don't have an idea how to do it. Will discuss that for 2.5 at least.
icinga-migration
commented
8 years ago Updated by gbeutner on 2016-01-13 09:31:16 +00:00
icinga-migration
commented
8 years ago Updated by Anonymous on 2016-01-13 09:31:24 +00:00
Applied in changeset 7f4af89a56d29812efbe3580799259108dc87484.
icinga-migration
commented
8 years ago Updated by gbeutner on 2016-01-13 09:31:45 +00:00
Please re-test this against the current Git master.
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-01-13 09:55:09 +00:00
The thread count stays the same after stopping the requests. Needs further investigation.
$ for i in {1..100000};do curl -k https://root:icinga@localhost:5665/v1/objects/hosts; done
mbmif ~ # ps M `pidof icinga2` | wc -l | xargs
270Updated by mfriedrich on 2016-01-22 15:58:53 +00:00
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-01-25 10:34:21 +00:00
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-01-27 09:34:50 +00:00
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-01-28 14:48:37 +00:00
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-01-28 14:48:50 +00:00
icinga-migration
commented
8 years ago Updated by Anonymous on 2016-02-03 09:05:04 +00:00
Applied in changeset 2dc385e31ba7f3dd217d3c36c7926017697bd079.
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-02-03 11:38:16 +00:00
There's still an existing reference to the HttpServerConnection object.
for i in {1..100000};do curl -k https://root:icinga@localhost:5665/v1/objects/hosts >/dev/null 2>&1; done
watch -n1 ps -o nlwp `pidof icinga2`[2016-02-03 10:36:23 +0000] information/HttpServerConnection: Request: GET /v1/objects/hosts (root)
[New Thread 0x7fffda780700 (LWP 23665)]
[Thread 0x7fffdaa4b700 (LWP 23649) exited]
Detaching after fork from child process 23666.
[Switching to Thread 0x7fffdaa0a700 (LWP 23650)]
Breakpoint 2, icinga::TlsStream::OnEvent (this=0x7fffe8004bc0, revents=4) at /root/coding/icinga2/lib/base/tlsstream.cpp:132
132 boost::mutex::scoped_lock lock(m_Mutex);
(gdb) bt
#0 icinga::TlsStream::OnEvent (this=0x7fffe8004bc0, revents=4) at /root/coding/icinga2/lib/base/tlsstream.cpp:132
#1 0x00007ffff70aa535 in icinga::SocketEvents::ThreadProc (tid=0) at /root/coding/icinga2/lib/base/socketevents.cpp:159
#2 0x00007ffff7093f67 in boost::_bi::list1 >::operator() (this=0x7fffe8004f30, f=@0x7fffe8004f28: 0x7ffff70a9d56 , a=...)at /usr/include/boost/bind/bind.hpp:253
#3 0x00007ffff7093f1b in boost::_bi::bind_t > >::operator() (this=0x7fffe8004f28) at /usr/include/boost/bind/bind_template.hpp:20
#4 0x00007ffff7093ee0 in boost::detail::thread_data > > >::run (this=0x7fffe8004d70) at /usr/include/boost/thread/detail/thread.hpp:117
#5 0x00007ffff7bd124a in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:165
#6 0x00007ffff3dd1dc5 in start_thread (arg=0x7fffdaa0a700) at pthread_create.c:308
#7 0x00007ffff40dc21d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113(gdb) c Continuing. Detaching after fork from child process 23667. [2016-02-03 10:36:37 +0000] warning/PluginCheckTask: Check command for object 'e4e752bf970b!ping6' (PID: 23664, arguments: '/usr/lib/nagios/plugins/check_ping' '-6' '-H' '::1' '-c' '200,15' '-w' '100,5') terminated with exit code 128, output: execvpe(/usr/lib/nagios/plugins/check_ping) failed: No such file or directory
Breakpoint 2, icinga::TlsStream::OnEvent (this=0x7fffe8004bc0, revents=1) at /root/coding/icinga2/lib/base/tlsstream.cpp:132 132 boost::mutex::scoped_lock lock(m_Mutex); (gdb) bt
#0 icinga::TlsStream::OnEvent (this=0x7fffe8004bc0, revents=1) at /root/coding/icinga2/lib/base/tlsstream.cpp:132
#1 0x00007ffff70aa535 in icinga::SocketEvents::ThreadProc (tid=0) at /root/coding/icinga2/lib/base/socketevents.cpp:159
#2 0x00007ffff7093f67 in boost::_bi::list1 >::operator() (this=0x7fffe8004f30, f=@0x7fffe8004f28: 0x7ffff70a9d56 , a=...)at /usr/include/boost/bind/bind.hpp:253
#3 0x00007ffff7093f1b in boost::_bi::bind_t > >::operator() (this=0x7fffe8004f28) at /usr/include/boost/bind/bind_template.hpp:20
#4 0x00007ffff7093ee0 in boost::detail::thread_data > > >::run (this=0x7fffe8004d70) at /usr/include/boost/thread/detail/thread.hpp:117
#5 0x00007ffff7bd124a in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:165
#6 0x00007ffff3dd1dc5 in start_thread (arg=0x7fffdaa0a700) at pthread_create.c:308
#7 0x00007ffff40dc21d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113(gdb) c Continuing. [New Thread 0x7fffdaa4b700 (LWP 23668)] Detaching after fork from child process 23669. [2016-02-03 10:36:37 +0000] information/Application: We jumped forward in time: 26.6704 seconds [2016-02-03 10:36:43 +0000] warning/ThreadPool: Event call took user:0s, system:0s, wait:6.00955s, minor_faults:8, major_faults:0, voluntary_csw:7, involuntary_csw:1 [2016-02-03 10:36:43 +0000] warning/ThreadPool: Event call took user:0s, system:0s, wait:12.5225s, minor_faults:8, major_faults:0, voluntary_csw:28, involuntary_csw:0
Breakpoint 1, icinga::TlsStream::OnEvent (this=0x7fffe8004bc0, revents=1) at /root/coding/icinga2/lib/base/tlsstream.cpp:209 209 Close(); (gdb) info b Num Type Disp Enb Address What 1 breakpoint keep y 0x00007ffff70db2a3 in icinga::TlsStream::OnEvent(int) at /root/coding/icinga2/lib/base/tlsstream.cpp:209 breakpoint already hit 1 time 2 breakpoint keep y 0x00007ffff70daf37 in icinga::TlsStream::OnEvent(int) at /root/coding/icinga2/lib/base/tlsstream.cpp:132 breakpoint already hit 6 times 3 breakpoint keep y 0x00007ffff70b6c0e in icinga::Stream::Close() at /root/coding/icinga2/lib/base/stream.cpp:81 breakpoint already hit 6 times 4 breakpoint keep y 0x00007ffff60a9ecb in icinga::HttpServerConnection::DataAvailableHandler() at /root/coding/icinga2/lib/remote/httpserverconnection.cpp:199 breakpoint already hit 3 times (gdb) s icinga::TlsStream::Close (this=0x7fffe8004bc0) at /root/coding/icinga2/lib/base/tlsstream.cpp:324 324 CloseInternal(false); (gdb) s icinga::TlsStream::CloseInternal (this=0x7fffe8004bc0, inDestructor=false) at /root/coding/icinga2/lib/base/tlsstream.cpp:329 329 if (m_Eof) (gdb) l 324 CloseInternal(false); 325 } 326 327 void TlsStream::CloseInternal(bool inDestructor) 328 { 329 if (m_Eof) 330 return; 331 332 m_Eof = true; 333 (gdb) n 332 m_Eof = true; (gdb) 334 if (!inDestructor) (gdb) 335 SignalDataAvailable(); (gdb)
Breakpoint 4, icinga::HttpServerConnection::DataAvailableHandler (this=0x7fffe8008470) at /root/coding/icinga2/lib/remote/httpserverconnection.cpp:199 199 bool close = false; (gdb) p m_Eof No symbol "m_Eof" in current context. (gdb) up
#1 0x00007ffff60ac480 in boost::_mfi::mf0::call > (this=0x7fffe8004af0, u=(boost::intrusive_ptr) 0x7fffe8008470)at /usr/include/boost/bind/mem_fn_template.hpp:40 40 BOOST_MEM_FN_RETURN (get_pointer(u)->*f_)(); (gdb) down
#0 icinga::HttpServerConnection::DataAvailableHandler (this=0x7fffe8008470) at /root/coding/icinga2/lib/remote/httpserverconnection.cpp:199199 bool close = false; (gdb) p *m_Stream.px $10 = {
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-02-03 11:38:23 +00:00
RegisterDataAvailableHandler, OnDataAvailable()
The older fix works on OSX (boost 1.59+) but not RHEL7 (boost 1.53)
[root@e4e752bf970b coding]# vim test.cpp
#include
#include
#include
#include
class Object
{
public:
void f(void)
{ }
int m_Refs;
};
void intrusive_ptr_add_ref(Object *object)
{
std::cout << "add_ref" << std::endl;
object->m_Refs++;
}
void intrusive_ptr_release(Object *object)
{
std::cout << "release" << std::endl;
object->m_Refs--;
}
boost::signals2::signal s;
int main(void)
{
boost::intrusive_ptr p = new Object();
s.connect(boost::bind(&Object::f, p));
p.reset();
s();
s.disconnect_all_slots();
// _exit(0);
}
[root@e4e752bf970b coding]# g++ -g -o test test.cpp
[root@e4e752bf970b coding]# ./test | grep release | wc -l
16
[root@e4e752bf970b coding]# ./test | grep add_ref | wc -l
17Updated by mfriedrich on 2016-02-03 11:54:18 +00:00
http://stackoverflow.com/questions/2049291/force-deletion-of-slot-in-boostsignals2
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-02-03 12:48:04 +00:00
Using the fix from https://github.com/Icinga/icinga2/commit/caf3380e73558c7f67bb5deb0d1c88edcaad0900
Every 1.0s: ps -o nlwp 10491 Wed Feb 3 13:47:40 2016
NLWP
27Updated by mfriedrich on 2016-02-03 12:56:37 +00:00
Affects boost versions > 1.49 and < 1.59 (e.g. RHEL 7).
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-02-03 12:56:49 +00:00
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-02-03 14:13:58 +00:00
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-02-04 09:08:35 +00:00
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-02-04 12:43:39 +00:00
https://svn.boost.org/trac/boost/ticket/8533 https://github.com/boostorg/signals2/commit/c22312b86c906368eb2ed0d5e95bfcf467d69785
icinga-migration
commented
8 years ago Updated by mfriedrich on 2016-02-08 08:59:32 +00:00
icinga-migration
commented
8 years ago Updated by gbeutner on 2016-02-23 09:58:18 +00:00
This issue has been migrated from Redmine: https://dev.icinga.com/issues/10655
Created by vvv on 2015-11-17 01:30:28 +00:00
Assignee: mfriedrich Status: Resolved (closed on 2016-02-03 12:56:49 +00:00) Target Version: 2.4.2 Last Update: 2016-02-23 09:58:18 +00:00 (in Redmine)
API Query opens threads but the thread does not die after the api query is done.
Checking for icinga2 threads:
icinga@icinga-np:/home/vvv$ for i in {1..100000}; do ps -eLf | grep "no-stack" | wc -l; sleep 5; done 17 17 17 17 17 17 17 22 32 42 52 62 72 82 84 84 84 84 84 84 84 84 84 84 84 84 84 84 84 84 84 84 84 84 84 84 84 84 84 85 85 85 85 85 85 85 86 86 86 86 86 86 86 86 86 86 87 87 87 87 87 87 91 94 97 100 103 107 110 113 116 119 123
Query used for API
VVV-MacBook-Pro:~ vvv$ for i in {1..100000};do curl -k https://root:icinga@icinga-np:5665/v1/objects/hosts?host=icinga-np; done
After killing the continuous curl call the thread count went steady at 123.
Changesets
2016-01-13 09:30:38 +00:00 by (unknown) 7f4af89a56d29812efbe3580799259108dc87484
2016-02-03 08:54:31 +00:00 by (unknown) 2dc385e31ba7f3dd217d3c36c7926017697bd079
2016-02-03 11:58:49 +00:00 by (unknown) cfe984950b4aae3f87533cf3765fae80b43a31cd
2016-02-03 12:02:49 +00:00 by (unknown) caf3380e73558c7f67bb5deb0d1c88edcaad0900
2016-02-23 08:38:05 +00:00 by (unknown) b6a799d53e66f2c7d5478b6a2ca0615a407916a5
2016-02-23 08:38:23 +00:00 by (unknown) 9a0107d360e335a57b1535231bf9591980f7e60f
2016-02-23 08:38:24 +00:00 by (unknown) 8af6ad74c3b4a9ae740de6bb3063ca3a7ba529f1
Relations: