Skydive in Docker container with dumb-init - zombie reaping problem

[mnasiadka]

Hello, In OpenStack Kolla we use skydive_agent in Docker container with dumb-init as the init system. Just after starting skydive_agent I can see a lot of ovs-ofctl zombie processes, which get killed after stopping skydive agent.

Skydive version: 0.21.0

Skydive config file:

Skydive agent config file

auth: analyzer_username: admin analyzer_password: [masked]

logging: level: INFO backends:

• file file: path: /var/log/kolla/skydive/skydive-agent.log

etcd: servers:

• http://172.17.1.101:2379
• http://172.17.1.102:2379
• http://172.17.1.103:2379

analyzers:

• 172.17.1.101:8085
• 172.17.1.102:8085
• 172.17.1.103:8085

agent: listen: 172.17.1.101:8090 flow: probes:

• gopacket
• ovssflow topology: probes:
• netlink
• netns
• neutron

• ovsdb

  neutron: auth_url: http://172.17.1.100:5000/v3 username: admin password: [masked] tenant_name: admin region_name: RegionOne domain_name: Default endpoint_type: internal

netns: run_path: /host/run

flow: expire: 600 update: 60

ovs: ovsdb: tcp://127.0.0.1:6640 oflow: enable: true

Skydive agent log: 2018-12-19T12:34:44.659Z INFO agent/agent.go:46 glob..func1 kolla-1: Skydive Agent 0.21.0 starting... 2018-12-19T12:34:44.660Z INFO http/server.go:109 (Server).Listen kolla-1: Listening on socket 172.17.1.101:8090 2018-12-19T12:34:44.677Z INFO agent/probes.go:49 NewTopologyProbeBundleFromConfig kolla-1: Topology probes: [netlink netns neutron ovsdb] 2018-12-19T12:34:44.679Z INFO ovsdb/ovs_of.go:893 NewOvsOfProbe kolla-1: Adding OVS probe on 127.0.0.1:6640 2018-12-19T12:34:44.680Z INFO probes/probes.go:67 NewFlowProbeBundle kolla-1: Flow probes: [pcapsocket ovssflow sflow gopacket dpdk ebpf ovsmirror] 2018-12-19T12:34:44.680Z INFO probes/probes.go:117 NewFlowProbeBundle kolla-1: Not compiled with dpdk support, skipping it 2018-12-19T12:34:44.680Z INFO probes/probes.go:117 NewFlowProbeBundle kolla-1: Not compiled with ebpf support, skipping it 2018-12-19T12:34:44.704Z INFO probes/ovsmirror.go:427 (OvsMirrorProbesHandler).cleanupOvsMirrors kolla-1: OvsMirror cleanup previous mirrors 2018-12-19T12:34:44.710Z INFO ovs/ovsdb.go:311 (OvsMonitor).portAdded kolla-1: New port "phy-br-ex(4ab4f324-c003-4b8f-847e-046df788595f)" added 2018-12-19T12:34:44.710Z INFO ovs/ovsdb.go:311 (OvsMonitor).portAdded kolla-1: New port "br-int(0fbe5538-e423-4693-b50c-e01a927294d5)" added 2018-12-19T12:34:44.710Z INFO ovs/ovsdb.go:311 (OvsMonitor).portAdded kolla-1: New port "patch-int(929aefea-c984-4d2c-b96f-e73b12cc1c8e)" added 2018-12-19T12:34:44.710Z INFO ovs/ovsdb.go:311 (OvsMonitor).portAdded kolla-1: New port "patch-tun(1c50030c-302d-435d-afd0-2fc282a226a8)" added 2018-12-19T12:34:44.710Z INFO ovs/ovsdb.go:311 (OvsMonitor).portAdded kolla-1: New port "br-ex(527ab1c4-b332-4e69-906e-96574e3a821e)" added 2018-12-19T12:34:44.710Z INFO ovs/ovsdb.go:311 (OvsMonitor).portAdded kolla-1: New port "br-tun(65376a95-a4c8-48d9-bf5d-7053b1094c50)" added 2018-12-19T12:34:44.710Z INFO ovs/ovsdb.go:311 (OvsMonitor).portAdded kolla-1: New port "int-br-ex(650e331f-b2a5-4eb9-945a-cf62ee76c2d8)" added 2018-12-19T12:34:44.710Z INFO ovs/ovsdb.go:311 (OvsMonitor).portAdded kolla-1: New port "enp4s0f1(fef0887b-d743-4a78-9752-99d46e3161ef)" added 2018-12-19T12:34:44.710Z INFO ovs/ovsdb.go:261 (OvsMonitor).interfaceAdded kolla-1: New interface "patch-tun(f65959cd-b190-4c5d-ad8e-909c9c4de5be)" added 2018-12-19T12:34:44.711Z INFO ovs/ovsdb.go:261 (OvsMonitor).interfaceAdded kolla-1: New interface "br-ex(2f35107d-6e7e-4ea1-9dad-0ae03913a499)" added 2018-12-19T12:34:44.711Z INFO ovs/ovsdb.go:261 (OvsMonitor).interfaceAdded kolla-1: New interface "int-br-ex(fcd22f14-a9ce-44b9-bd17-116d3ef5163e)" added 2018-12-19T12:34:44.711Z INFO ovs/ovsdb.go:261 (OvsMonitor).interfaceAdded kolla-1: New interface "patch-int(51fb2070-fa6f-4a5c-b63f-66ea98b893fb)" added 2018-12-19T12:34:44.711Z INFO ovs/ovsdb.go:261 (OvsMonitor).interfaceAdded kolla-1: New interface "enp4s0f1(37420e6c-fb51-4330-9f5c-ea5f5389b207)" added 2018-12-19T12:34:44.711Z INFO ovs/ovsdb.go:261 (OvsMonitor).interfaceAdded kolla-1: New interface "br-tun(2b0db1a0-b63f-4de6-995a-f5c7773752e6)" added 2018-12-19T12:34:44.711Z INFO ovs/ovsdb.go:261 (OvsMonitor).interfaceAdded kolla-1: New interface "br-int(6e6fea23-108a-48fb-8864-5ac416915c3b)" added 2018-12-19T12:34:44.711Z INFO ovs/ovsdb.go:261 (OvsMonitor).interfaceAdded kolla-1: New interface "phy-br-ex(49ba1791-cf4e-4a7d-8933-0f7ed3af6621)" added 2018-12-19T12:34:44.711Z INFO ovs/ovsdb.go:210 (OvsMonitor).bridgeAdded kolla-1: New bridge "br-int(d5ea8493-c5cd-4df2-b984-8fc4f407b2b1)" added 2018-12-19T12:34:44.751Z WARN ovsdb/ovs_of.go:717 (BridgeOfProbe).monitorGroup kolla-1: OpenFlow 1.5 not supported on br-int 2018-12-19T12:34:44.751Z ERROR ovsdb/ovs_of.go:814 (OvsOfProbe).NewBridgeProbe kolla-1: Cannot add group probe on br-int - exit status 1 2018-12-19T12:34:44.751Z INFO ovs/ovsdb.go:210 (OvsMonitor).bridgeAdded kolla-1: New bridge "br-ex(08b7e248-de11-495f-836f-e2cded2d2125)" added 2018-12-19T12:34:44.777Z WARN ovsdb/ovs_of.go:717 (BridgeOfProbe).monitorGroup kolla-1: OpenFlow 1.5 not supported on br-ex 2018-12-19T12:34:44.777Z ERROR ovsdb/ovs_of.go:814 (OvsOfProbe).NewBridgeProbe kolla-1: Cannot add group probe on br-ex - exit status 1 2018-12-19T12:34:44.777Z INFO ovs/ovsdb.go:210 (OvsMonitor).bridgeAdded kolla-1: New bridge "br-tun(40b1a3ca-1e6a-4420-a8a6-b4899b6f62c1)" added 2018-12-19T12:34:44.777Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule d88bca7d-3560-5327-5c0d-f9e22347ccdd added 2018-12-19T12:34:44.797Z WARN ovsdb/ovs_of.go:717 (BridgeOfProbe).monitorGroup kolla-1: OpenFlow 1.5 not supported on br-tun 2018-12-19T12:34:44.797Z ERROR ovsdb/ovs_of.go:814 (OvsOfProbe).NewBridgeProbe kolla-1: Cannot add group probe on br-tun - exit status 1 2018-12-19T12:34:44.797Z INFO agent/agent.go:56 glob..func1 kolla-1: Skydive Agent started 2018-12-19T12:34:44.797Z INFO websocket/client.go:475 (Client).Connect kolla-1: Connecting to ws://172.17.1.103:8085/ws/agent/topology 2018-12-19T12:34:44.797Z INFO websocket/client.go:475 (Client).Connect kolla-1: Connecting to ws://172.17.1.102:8085/ws/agent/topology 2018-12-19T12:34:44.797Z INFO websocket/client.go:475 (Client).Connect kolla-1: Connecting to ws://172.17.1.101:8085/ws/agent/topology 2018-12-19T12:34:44.798Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule d0aa2f37-f491-57d1-79a4-8fd1a81a16c5 added 2018-12-19T12:34:44.800Z INFO websocket/client.go:502 (Client).Connect kolla-1: Connected to ws://172.17.1.101:8085/ws/agent/topology 2018-12-19T12:34:44.800Z INFO agent/topology_forwarder.go:61 (TopologyForwarder).OnNewMaster kolla-1: Using 172.17.1.101:8085 as master of topology forwarder 2018-12-19T12:34:44.800Z INFO websocket/client.go:502 (Client).Connect kolla-1: Connected to ws://172.17.1.102:8085/ws/agent/topology 2018-12-19T12:34:44.800Z INFO agent/topology_forwarder.go:42 (TopologyForwarder).triggerResync kolla-1: Start a re-sync for kolla-1 2018-12-19T12:34:44.800Z INFO websocket/client.go:502 (Client).Connect kolla-1: Connected to ws://172.17.1.103:8085/ws/agent/topology 2018-12-19T12:34:44.806Z INFO probes/ovsmirror.go:427 (OvsMirrorProbesHandler).cleanupOvsMirrors kolla-1: OvsMirror cleanup previous mirrors 2018-12-19T12:34:44.815Z INFO ovs/ovsdb.go:300 (OvsMonitor).portUpdated kolla-1: Port "patch-tun(1c50030c-302d-435d-afd0-2fc282a226a8)" updated 2018-12-19T12:34:44.816Z INFO ovs/ovsdb.go:300 (OvsMonitor).portUpdated kolla-1: Port "br-ex(527ab1c4-b332-4e69-906e-96574e3a821e)" updated 2018-12-19T12:34:44.816Z INFO ovs/ovsdb.go:300 (OvsMonitor).portUpdated kolla-1: Port "br-tun(65376a95-a4c8-48d9-bf5d-7053b1094c50)" updated 2018-12-19T12:34:44.816Z INFO ovs/ovsdb.go:300 (OvsMonitor).portUpdated kolla-1: Port "int-br-ex(650e331f-b2a5-4eb9-945a-cf62ee76c2d8)" updated 2018-12-19T12:34:44.816Z INFO ovs/ovsdb.go:300 (OvsMonitor).portUpdated kolla-1: Port "enp4s0f1(fef0887b-d743-4a78-9752-99d46e3161ef)" updated 2018-12-19T12:34:44.816Z INFO ovs/ovsdb.go:300 (OvsMonitor).portUpdated kolla-1: Port "phy-br-ex(4ab4f324-c003-4b8f-847e-046df788595f)" updated 2018-12-19T12:34:44.817Z INFO ovs/ovsdb.go:300 (OvsMonitor).portUpdated kolla-1: Port "br-int(0fbe5538-e423-4693-b50c-e01a927294d5)" updated 2018-12-19T12:34:44.817Z INFO ovs/ovsdb.go:300 (OvsMonitor).portUpdated kolla-1: Port "patch-int(929aefea-c984-4d2c-b96f-e73b12cc1c8e)" updated 2018-12-19T12:34:44.817Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "br-ex(2f35107d-6e7e-4ea1-9dad-0ae03913a499)" updated 2018-12-19T12:34:44.817Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "int-br-ex(fcd22f14-a9ce-44b9-bd17-116d3ef5163e)" updated 2018-12-19T12:34:44.818Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "patch-int(51fb2070-fa6f-4a5c-b63f-66ea98b893fb)" updated 2018-12-19T12:34:44.818Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "enp4s0f1(37420e6c-fb51-4330-9f5c-ea5f5389b207)" updated 2018-12-19T12:34:44.818Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "br-tun(2b0db1a0-b63f-4de6-995a-f5c7773752e6)" updated 2018-12-19T12:34:44.818Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "br-int(6e6fea23-108a-48fb-8864-5ac416915c3b)" updated 2018-12-19T12:34:44.818Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "phy-br-ex(49ba1791-cf4e-4a7d-8933-0f7ed3af6621)" updated 2018-12-19T12:34:44.819Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "patch-tun(f65959cd-b190-4c5d-ad8e-909c9c4de5be)" updated 2018-12-19T12:34:44.819Z INFO ovs/ovsdb.go:199 (OvsMonitor).bridgeUpdated kolla-1: Bridge "br-int(d5ea8493-c5cd-4df2-b984-8fc4f407b2b1)" updated 2018-12-19T12:34:44.820Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 71ba9d7b-aac6-5dbe-4e6b-6b82d2f64d03 added 2018-12-19T12:34:44.824Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 97655ad9-b195-59fa-7a00-5d710c66130d added 2018-12-19T12:34:44.826Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule faa04bc3-71f5-5820-783f-ea215feaac6f added 2018-12-19T12:34:44.837Z WARN ovsdb/ovs_of.go:717 (BridgeOfProbe).monitorGroup kolla-1: OpenFlow 1.5 not supported on br-int 2018-12-19T12:34:44.837Z INFO ovs/ovsdb.go:199 (OvsMonitor).bridgeUpdated kolla-1: Bridge "br-ex(08b7e248-de11-495f-836f-e2cded2d2125)" updated 2018-12-19T12:34:44.856Z WARN ovsdb/ovs_of.go:717 (BridgeOfProbe).monitorGroup kolla-1: OpenFlow 1.5 not supported on br-ex 2018-12-19T12:34:44.856Z INFO ovs/ovsdb.go:199 (OvsMonitor).bridgeUpdated kolla-1: Bridge "br-tun(40b1a3ca-1e6a-4420-a8a6-b4899b6f62c1)" updated 2018-12-19T12:34:44.856Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 3e38742f-d9ab-5867-7dc6-a70695051e71 added 2018-12-19T12:34:44.885Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 4df00e64-e1fd-5ac1-6e77-f4d479888c77 added 2018-12-19T12:34:44.887Z WARN ovsdb/ovs_of.go:717 (BridgeOfProbe).monitorGroup kolla-1: OpenFlow 1.5 not supported on br-tun 2018-12-19T12:34:44.890Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 63889baa-6342-558e-6711-0adf6f0308d1 added 2018-12-19T12:34:44.916Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 7acaa5c6-48df-574a-4ec3-bdc26216fcd7 added 2018-12-19T12:34:44.921Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 04c16d29-a231-5b10-5827-79dd135b55d8 added 2018-12-19T12:34:44.941Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 4ecfd4bb-fc7e-5428-6312-1d57c1546f09 added 2018-12-19T12:34:44.946Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 19d86c08-ad6a-5441-6a87-82d00df6ba75 added 2018-12-19T12:34:44.948Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule c6e70d18-e5bb-5263-62bf-f3b8cee08dea added 2018-12-19T12:34:44.967Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 0f138c9c-adc2-58b2-73d9-b630bf1bbe16 added 2018-12-19T12:34:44.969Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 2311b2f7-04de-5313-6af3-84a68612519a added 2018-12-19T12:34:44.972Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 24b6751a-ad6f-5c91-7541-311e57377e08 added 2018-12-19T12:34:44.987Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule c8863065-d66e-5f33-6eb6-8784dafb2bdd added 2018-12-19T12:34:44.988Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 6c35dc42-c6ce-5ef2-55c6-1d036e265421 added 2018-12-19T12:34:44.994Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 001e1917-bc2e-5314-7e98-5ebee93b28fe added 2018-12-19T12:34:45.007Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 10fa6ff8-e55d-5acc-5c19-a8912ebd89f2 added 2018-12-19T12:34:45.007Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule d3e85d76-1ca4-5628-7687-27f06e1519db added 2018-12-19T12:34:45.015Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 6efc5af9-7568-5c23-49b4-c4b5caf5f3dc added 2018-12-19T12:34:45.025Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 8d70b99e-013c-5341-6599-80346bd7f2fa added 2018-12-19T12:34:45.032Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 4b2db093-e9de-5384-7855-5b55ea372fae added 2018-12-19T12:34:45.045Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule a5316efc-9fc5-5e2c-6e84-8958eb84d712 added 2018-12-19T12:34:45.065Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 68e4493f-3211-54a1-434e-e598d15bd309 added 2018-12-19T12:34:45.084Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 25f92847-10b4-54c4-692d-7497311d3456 added 2018-12-19T12:34:45.101Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule a4517dd9-9b46-5048-4ffe-ca7aebaad388 added 2018-12-19T12:34:45.122Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule 77a23f55-c73f-53bc-5ee2-6ffb7473c420 added 2018-12-19T12:34:45.145Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule a9a1be17-f8bd-5b9f-45c5-ee5f672f4b19 added 2018-12-19T12:34:45.168Z INFO ovsdb/ovs_of.go:497 (BridgeOfProbe).addRule kolla-1: New rule a7326d69-1548-580d-70ed-882f6be39bc4 added 2018-12-19T12:34:48.609Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "br-ex(2f35107d-6e7e-4ea1-9dad-0ae03913a499)" updated 2018-12-19T12:34:48.609Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "enp4s0f1(37420e6c-fb51-4330-9f5c-ea5f5389b207)" updated 2018-12-19T12:34:48.609Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "phy-br-ex(49ba1791-cf4e-4a7d-8933-0f7ed3af6621)" updated 2018-12-19T12:34:48.610Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "int-br-ex(fcd22f14-a9ce-44b9-bd17-116d3ef5163e)" updated 2018-12-19T12:34:48.611Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "enp4s0f1(37420e6c-fb51-4330-9f5c-ea5f5389b207)" updated 2018-12-19T12:34:48.611Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "phy-br-ex(49ba1791-cf4e-4a7d-8933-0f7ed3af6621)" updated 2018-12-19T12:34:48.612Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "int-br-ex(fcd22f14-a9ce-44b9-bd17-116d3ef5163e)" updated 2018-12-19T12:34:48.612Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "br-ex(2f35107d-6e7e-4ea1-9dad-0ae03913a499)" updated 2018-12-19T12:34:53.611Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "br-ex(2f35107d-6e7e-4ea1-9dad-0ae03913a499)" updated 2018-12-19T12:34:53.611Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "enp4s0f1(37420e6c-fb51-4330-9f5c-ea5f5389b207)" updated 2018-12-19T12:34:53.612Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "phy-br-ex(49ba1791-cf4e-4a7d-8933-0f7ed3af6621)" updated 2018-12-19T12:34:53.612Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "int-br-ex(fcd22f14-a9ce-44b9-bd17-116d3ef5163e)" updated 2018-12-19T12:34:53.614Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "br-ex(2f35107d-6e7e-4ea1-9dad-0ae03913a499)" updated 2018-12-19T12:34:53.614Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "enp4s0f1(37420e6c-fb51-4330-9f5c-ea5f5389b207)" updated 2018-12-19T12:34:53.615Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "phy-br-ex(49ba1791-cf4e-4a7d-8933-0f7ed3af6621)" updated 2018-12-19T12:34:53.615Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "int-br-ex(fcd22f14-a9ce-44b9-bd17-116d3ef5163e)" updated 2018-12-19T12:34:58.614Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "br-ex(2f35107d-6e7e-4ea1-9dad-0ae03913a499)" updated 2018-12-19T12:34:58.615Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "enp4s0f1(37420e6c-fb51-4330-9f5c-ea5f5389b207)" updated 2018-12-19T12:34:58.616Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "phy-br-ex(49ba1791-cf4e-4a7d-8933-0f7ed3af6621)" updated 2018-12-19T12:34:58.616Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "int-br-ex(fcd22f14-a9ce-44b9-bd17-116d3ef5163e)" updated 2018-12-19T12:34:58.617Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "br-ex(2f35107d-6e7e-4ea1-9dad-0ae03913a499)" updated 2018-12-19T12:34:58.617Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "enp4s0f1(37420e6c-fb51-4330-9f5c-ea5f5389b207)" updated 2018-12-19T12:34:58.617Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "phy-br-ex(49ba1791-cf4e-4a7d-8933-0f7ed3af6621)" updated 2018-12-19T12:34:58.618Z INFO ovs/ovsdb.go:250 (OvsMonitor).interfaceUpdated kolla-1: Interface "int-br-ex(fcd22f14-a9ce-44b9-bd17-116d3ef5163e)" updated 2018-12-19T12:35:01.817Z ERROR ovsdb/ovs_of.go:645 (BridgeOfProbe).monitor.func1 kolla-1: Error while monitoring br-tun@127.0.0.1:6640: Cannot launch ovs-ofctl dump-flows on br-tun@127.0.0.1:6640 with filter table=2,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00: signal: terminated 2018-12-19T12:35:01.817Z ERROR ovsdb/ovs_of.go:645 (BridgeOfProbe).monitor.func1 kolla-1: Error while monitoring br-int@127.0.0.1:6640: Cannot launch ovs-ofctl dump-flows on br-int@127.0.0.1:6640 with filter table=0,in_port=1,dl_src=fa:16:3f:02:c2:df: signal: terminated 2018-12-19T12:35:01.818Z ERROR ovsdb/ovs_of.go:645 (BridgeOfProbe).monitor.func1 kolla-1: Error while monitoring br-ex@127.0.0.1:6640: Cannot launch ovs-ofctl dump-flows on br-ex@127.0.0.1:6640 with filter table=1: signal: terminated 2018-12-19T12:35:01.820Z INFO probes/ovsmirror.go:427 (*OvsMirrorProbesHandler).cleanupOvsMirrors kolla-1: OvsMirror cleanup previous mirrors 2018-12-19T12:35:01.820Z WARN ovs/ovsdb.go:152 Notifier.Disconnected kolla-1: Disconnected from OVSDB 2018-12-19T12:35:02.681Z INFO agent/agent.go:63 glob..func1 kolla-1: Skydive Agent stopped.

Docker log: [dumb-init] Child spawned with PID 7. [dumb-init] Unable to attach to controlling tty (errno=25 Not a tty). [dumb-init] setsid complete.

• sudo -E kolla_set_configs INFO:main:Loading config file at /var/lib/kolla/config_files/config.json INFO:main:Validating config file INFO:main:Kolla config strategy set to: COPY_ALWAYS INFO:main:Copying service configuration files INFO:main:Deleting /etc/skydive/skydive.conf INFO:main:Copying /var/lib/kolla/config_files/skydive.conf to /etc/skydive/skydive.conf INFO:main:Setting permission for /etc/skydive/skydive.conf INFO:main:Writing out command to execute INFO:main:Setting permission for /var/log/kolla/skydive INFO:main:Setting permission for /var/log/kolla/skydive/skydive-agent.log INFO:main:Setting permission for /var/log/kolla/skydive/skydive-analyzer.log ++ cat /run_command
• CMD='skydive agent --conf /etc/skydive/skydive.conf --listen=172.17.1.101:8090'
• ARGS=
• [[ ! -n '' ]]
• . kolla_extend_start ++ [[ ! -d /var/log/kolla/skydive ]] +++ stat -c %a /var/log/kolla/skydive ++ [[ 755 != \7\5\5 ]]
• echo 'Running command: '\''skydive agent --conf /etc/skydive/skydive.conf --listen=172.17.1.101:8090'\'''
• exec skydive agent --conf /etc/skydive/skydive.conf --listen=172.17.1.101:8090 Running command: 'skydive agent --conf /etc/skydive/skydive.conf --listen=172.17.1.101:8090' Config value 'auth.analyzer_username' is now deprecated. Please use 'agent.auth.cluster.username' instead Config value 'auth.analyzer_password' is now deprecated. Please use 'agent.auth.cluster.password' instead [dumb-init] Received signal 15. [dumb-init] Forwarded signal 15 to children. [dumb-init] Received signal 17. [dumb-init] A child with PID 297 exited with exit status 1. [dumb-init] A child with PID 1620 exited with exit status 1. [dumb-init] A child with PID 1630 exited with exit status 1. [dumb-init] A child with PID 1631 exited with exit status 1. [dumb-init] A child with PID 1653 exited with exit status 1. [dumb-init] A child with PID 1722 exited with exit status 1. [dumb-init] A child with PID 1731 exited with exit status 1. [dumb-init] A child with PID 1791 exited with exit status 1. [dumb-init] A child with PID 1833 exited with exit status 1. [dumb-init] A child with PID 1910 exited with exit status 1. [dumb-init] A child with PID 1956 exited with exit status 1. [dumb-init] A child with PID 2030 exited with exit status 1. [dumb-init] A child with PID 2035 exited with exit status 1. [dumb-init] A child with PID 2067 exited with exit status 1. [dumb-init] A child with PID 2072 exited with exit status 1. [dumb-init] A child with PID 2096 exited with exit status 1. [dumb-init] A child with PID 2119 exited with exit status 1. [dumb-init] A child with PID 2216 exited with exit status 1. [dumb-init] A child with PID 2262 exited with exit status 1. [dumb-init] A child with PID 1239 exited with exit status 1. [dumb-init] A child with PID 1252 exited with exit status 1. [dumb-init] A child with PID 1269 exited with exit status 1. [dumb-init] A child with PID 1320 exited with exit status 1. [dumb-init] A child with PID 1471 exited with exit status 1. [dumb-init] A child with PID 2518 exited with exit status 1. [dumb-init] A child with PID 2540 exited with exit status 1. [dumb-init] A child with PID 2612 exited with exit status 1. [dumb-init] A child with PID 2640 exited with exit status 1. [dumb-init] A child with PID 2671 exited with exit status 1. [dumb-init] A child with PID 2776 exited with exit status 1. [dumb-init] A child with PID 139 exited with exit status 1. [dumb-init] A child with PID 156 exited with exit status 1. [dumb-init] A child with PID 159 exited with exit status 1. [dumb-init] A child with PID 401 exited with exit status 1. [dumb-init] A child with PID 714 exited with exit status 1. [dumb-init] A child with PID 2098 exited with exit status 1. [dumb-init] A child with PID 2226 exited with exit status 1. [dumb-init] A child with PID 2247 exited with exit status 1. [dumb-init] A child with PID 2517 exited with exit status 1. [dumb-init] A child with PID 2621 exited with exit status 1. [dumb-init] A child with PID 2667 exited with exit status 1. [dumb-init] A child with PID 175 exited with exit status 1. [dumb-init] A child with PID 235 exited with exit status 1. [dumb-init] A child with PID 271 exited with exit status 1. [dumb-init] A child with PID 340 exited with exit status 1. [dumb-init] A child with PID 343 exited with exit status 1. [dumb-init] A child with PID 365 exited with exit status 1. [dumb-init] A child with PID 384 exited with exit status 1. [dumb-init] A child with PID 388 exited with exit status 1. [dumb-init] A child with PID 1230 exited with exit status 1. [dumb-init] A child with PID 1297 exited with exit status 1. [dumb-init] A child with PID 1299 exited with exit status 1. [dumb-init] A child with PID 1329 exited with exit status 1. [dumb-init] A child with PID 1330 exited with exit status 1. [dumb-init] A child with PID 1379 exited with exit status 1. [dumb-init] A child with PID 1386 exited with exit status 1. [dumb-init] A child with PID 1561 exited with exit status 1. [dumb-init] A child with PID 2129 exited with exit status 1. [dumb-init] A child with PID 2156 exited with exit status 1. [dumb-init] A child with PID 2187 exited with exit status 1. [dumb-init] A child with PID 2192 exited with exit status 1. [dumb-init] A child with PID 2275 exited with exit status 1. [dumb-init] A child with PID 2400 exited with exit status 1. [dumb-init] A child with PID 792 exited with exit status 1. [dumb-init] A child with PID 1065 exited with exit status 1. [dumb-init] A child with PID 1081 exited with exit status 1. [dumb-init] A child with PID 1088 exited with exit status 1. [dumb-init] A child with PID 127 exited with exit status 1. [dumb-init] A child with PID 251 exited with exit status 1. [dumb-init] A child with PID 280 exited with exit status 1. [dumb-init] A child with PID 284 exited with exit status 1. [dumb-init] A child with PID 410 exited with exit status 1. [dumb-init] A child with PID 435 exited with exit status 1. [dumb-init] A child with PID 2167 exited with exit status 1. [dumb-init] A child with PID 2176 exited with exit status 1. [dumb-init] A child with PID 2276 exited with exit status 1. [dumb-init] A child with PID 2334 exited with exit status 1. [dumb-init] A child with PID 2356 exited with exit status 1. [dumb-init] A child with PID 2377 exited with exit status 1. [dumb-init] A child with PID 2395 exited with exit status 1. [dumb-init] A child with PID 2423 exited with exit status 1. [dumb-init] A child with PID 2426 exited with exit status 1. [dumb-init] A child with PID 2457 exited with exit status 1. [dumb-init] A child with PID 2549 exited with exit status 1. [dumb-init] A child with PID 2590 exited with exit status 1. [dumb-init] A child with PID 2608 exited with exit status 1. [dumb-init] A child with PID 2634 exited with exit status 1. [dumb-init] A child with PID 484 exited with exit status 1. [dumb-init] A child with PID 572 exited with exit status 1. [dumb-init] A child with PID 621 exited with exit status 1. [dumb-init] A child with PID 674 exited with exit status 1. [dumb-init] A child with PID 682 exited with exit status 1. [dumb-init] A child with PID 737 exited with exit status 1. [dumb-init] A child with PID 493 exited with exit status 1. [dumb-init] A child with PID 553 exited with exit status 1. [dumb-init] A child with PID 584 exited with exit status 1. [dumb-init] A child with PID 585 exited with exit status 1. [dumb-init] A child with PID 1428 exited with exit status 1. [dumb-init] A child with PID 1528 exited with exit status 1. [dumb-init] A child with PID 1532 exited with exit status 1. [dumb-init] A child with PID 1591 exited with exit status 1. [dumb-init] A child with PID 1766 exited with exit status 1. [dumb-init] A child with PID 1783 exited with exit status 1. [dumb-init] A child with PID 1976 exited with exit status 1. [dumb-init] A child with PID 1978 exited with exit status 1. [dumb-init] A child with PID 2002 exited with exit status 1. [dumb-init] A child with PID 2008 exited with exit status 1. [dumb-init] A child with PID 2050 exited with exit status 1. [dumb-init] A child with PID 2447 exited with exit status 1. [dumb-init] A child with PID 2486 exited with exit status 1. [dumb-init] A child with PID 98 exited with exit status 1. [dumb-init] A child with PID 202 exited with exit status 1. [dumb-init] A child with PID 220 exited with exit status 1. [dumb-init] A child with PID 227 exited with exit status 1. [dumb-init] A child with PID 319 exited with exit status 1. [dumb-init] A child with PID 458 exited with exit status 1. [dumb-init] A child with PID 461 exited with exit status 1. [dumb-init] A child with PID 522 exited with exit status 1. [dumb-init] A child with PID 763 exited with exit status 1. [dumb-init] A child with PID 781 exited with exit status 1. [dumb-init] A child with PID 807 exited with exit status 1. [dumb-init] A child with PID 879 exited with exit status 1. [dumb-init] A child with PID 910 exited with exit status 1. [dumb-init] A child with PID 993 exited with exit status 1. [dumb-init] A child with PID 1131 exited with exit status 1. [dumb-init] A child with PID 1136 exited with exit status 1. [dumb-init] A child with PID 1676 exited with exit status 1. [dumb-init] A child with PID 1679 exited with exit status 1. [dumb-init] A child with PID 1682 exited with exit status 1. [dumb-init] A child with PID 1709 exited with exit status 1. [dumb-init] A child with PID 1769 exited with exit status 1. [dumb-init] A child with PID 1815 exited with exit status 1. [dumb-init] A child with PID 1825 exited with exit status 1. [dumb-init] A child with PID 1886 exited with exit status 1. [dumb-init] A child with PID 1918 exited with exit status 1. [dumb-init] A child with PID 1933 exited with exit status 1. [dumb-init] A child with PID 1947 exited with exit status 1. [dumb-init] A child with PID 1864 exited with exit status 1. [dumb-init] A child with PID 1879 exited with exit status 1. [dumb-init] A child with PID 2025 exited with exit status 1. [dumb-init] A child with PID 2579 exited with exit status 1. [dumb-init] A child with PID 2730 exited with exit status 1. [dumb-init] A child with PID 86 exited with exit status 1. [dumb-init] A child with PID 90 exited with exit status 1. [dumb-init] A child with PID 1227 exited with exit status 1. [dumb-init] A child with PID 1478 exited with exit status 1. [dumb-init] A child with PID 1492 exited with exit status 1. [dumb-init] A child with PID 1501 exited with exit status 1. [dumb-init] A child with PID 1540 exited with exit status 1. [dumb-init] A child with PID 1585 exited with exit status 1. [dumb-init] A child with PID 1650 exited with exit status 1. [dumb-init] A child with PID 1700 exited with exit status 1. [dumb-init] A child with PID 1739 exited with exit status 1. [dumb-init] A child with PID 1744 exited with exit status 1. [dumb-init] A child with PID 1798 exited with exit status 1. [dumb-init] A child with PID 1840 exited with exit status 1. [dumb-init] A child with PID 2127 exited with exit status 1. [dumb-init] A child with PID 2142 exited with exit status 1. [dumb-init] A child with PID 2213 exited with exit status 1. [dumb-init] A child with PID 2287 exited with exit status 1. [dumb-init] A child with PID 2303 exited with exit status 1. [dumb-init] A child with PID 2336 exited with exit status 1. [dumb-init] A child with PID 2364 exited with exit status 1. [dumb-init] A child with PID 2473 exited with exit status 1. [dumb-init] A child with PID 2522 exited with exit status 1. [dumb-init] A child with PID 2682 exited with exit status 1. [dumb-init] A child with PID 2696 exited with exit status 1. [dumb-init] A child with PID 2705 exited with exit status 1. [dumb-init] A child with PID 2722 exited with exit status 1. [dumb-init] A child with PID 2751 exited with exit status 1. [dumb-init] A child with PID 2759 exited with exit status 1. [dumb-init] A child with PID 2772 exited with exit status 1. [dumb-init] A child with PID 2788 exited with exit status 1. [dumb-init] A child with PID 1607 exited with exit status 1. [dumb-init] A child with PID 1857 exited with exit status 1. [dumb-init] A child with PID 1889 exited with exit status 1. [dumb-init] A child with PID 2080 exited with exit status 1. [dumb-init] A child with PID 830 exited with exit status 1. [dumb-init] A child with PID 882 exited with exit status 1. [dumb-init] A child with PID 925 exited with exit status 1. [dumb-init] A child with PID 972 exited with exit status 1. [dumb-init] A child with PID 1029 exited with exit status 1. [dumb-init] A child with PID 1041 exited with exit status 1. [dumb-init] A child with PID 1180 exited with exit status 1. [dumb-init] A child with PID 1203 exited with exit status 1. [dumb-init] A child with PID 1209 exited with exit status 1. [dumb-init] A child with PID 2239 exited with exit status 1. [dumb-init] A child with PID 2312 exited with exit status 1. [dumb-init] A child with PID 41 exited with exit status 1. [dumb-init] A child with PID 44 exited with exit status 1. [dumb-init] A child with PID 981 exited with exit status 1. [dumb-init] A child with PID 1017 exited with exit status 1. [dumb-init] A child with PID 1089 exited with exit status 1. [dumb-init] A child with PID 1413 exited with exit status 1. [dumb-init] A child with PID 1515 exited with exit status 1. [dumb-init] A child with PID 1930 exited with exit status 1. [dumb-init] A child with PID 1981 exited with exit status 1. [dumb-init] A child with PID 2325 exited with exit status 1. [dumb-init] A child with PID 2376 exited with exit status 1. [dumb-init] A child with PID 2425 exited with exit status 1. [dumb-init] A child with PID 2566 exited with exit status 1. [dumb-init] A child with PID 2572 exited with exit status 1. [dumb-init] A child with PID 645 exited with exit status 1. [dumb-init] A child with PID 666 exited with exit status 1. [dumb-init] A child with PID 703 exited with exit status 1. [dumb-init] A child with PID 731 exited with exit status 1. [dumb-init] A child with PID 822 exited with exit status 1. [dumb-init] A child with PID 903 exited with exit status 1. [dumb-init] A child with PID 1000 exited with exit status 1. [dumb-init] A child with PID 1158 exited with exit status 1. [dumb-init] A child with PID 1275 exited with exit status 1. [dumb-init] A child with PID 1283 exited with exit status 1. [dumb-init] A child with PID 1345 exited with exit status 1. [dumb-init] A child with PID 1371 exited with exit status 1. [dumb-init] A child with PID 1395 exited with exit status 1. [dumb-init] A child with PID 1421 exited with exit status 1. [dumb-init] A child with PID 1443 exited with exit status 1. [dumb-init] A child with PID 1445 exited with exit status 1. [dumb-init] A child with PID 1468 exited with exit status 1. [dumb-init] A child with PID 1562 exited with exit status 1. [dumb-init] A child with PID 1580 exited with exit status 1. [dumb-init] A child with PID 109 exited with exit status 1. [dumb-init] A child with PID 190 exited with exit status 1. [dumb-init] A child with PID 248 exited with exit status 1. [dumb-init] A child with PID 310 exited with exit status 1. [dumb-init] A child with PID 333 exited with exit status 1. [dumb-init] A child with PID 481 exited with exit status 1. [dumb-init] A child with PID 503 exited with exit status 1. [dumb-init] A child with PID 529 exited with exit status 1. [dumb-init] A child with PID 535 exited with exit status 1. [dumb-init] A child with PID 551 exited with exit status 1. [dumb-init] A child with PID 634 exited with exit status 1. [dumb-init] A child with PID 643 exited with exit status 1. [dumb-init] A child with PID 690 exited with exit status 1. [dumb-init] A child with PID 733 exited with exit status 1. [dumb-init] A child with PID 49 exited with exit status 1. [dumb-init] A child with PID 62 exited with exit status 1. [dumb-init] A child with PID 833 exited with exit status 1. [dumb-init] A child with PID 853 exited with exit status 1. [dumb-init] A child with PID 875 exited with exit status 1. [dumb-init] A child with PID 929 exited with exit status 1. [dumb-init] A child with PID 940 exited with exit status 1. [dumb-init] A child with PID 969 exited with exit status 1. [dumb-init] A child with PID 1030 exited with exit status 1. [dumb-init] A child with PID 1112 exited with exit status 1. [dumb-init] A child with PID 1148 exited with exit status 1. [dumb-init] A child with PID 1176 exited with exit status 1. [dumb-init] A child with PID 1182 exited with exit status 1. [dumb-init] A child with PID 1358 exited with exit status 1. [dumb-init] A child with PID 47 exited with exit status 1. [dumb-init] A child with PID 133 exited with exit status 1. [dumb-init] A child with PID 186 exited with exit status 1. [dumb-init] A child with PID 370 exited with exit status 1. [dumb-init] A child with PID 431 exited with exit status 1. [dumb-init] A child with PID 433 exited with exit status 1. [dumb-init] A child with PID 597 exited with exit status 1. [dumb-init] A child with PID 614 exited with exit status 1. [dumb-init] A child with PID 761 exited with exit status 1. [dumb-init] A child with PID 784 exited with exit status 1. [dumb-init] A child with PID 852 exited with exit status 1. [dumb-init] A child with PID 947 exited with exit status 1. [dumb-init] A child with PID 1059 exited with exit status 1. [dumb-init] A child with PID 1117 exited with exit status 1. [dumb-init] Received signal 17. [dumb-init] Received signal 17. [dumb-init] A child with PID 7 exited with exit status 0. [dumb-init] Forwarded signal 15 to children. [dumb-init] Child exited with status 0. Goodbye.

[lebauce]

@pierrecregut Any idea ?

[pierrecregut]

Indeed there is a Wait missing for launchContinuousOnSwitch. As the command is retried until the bridge is available this creates a lot of zombie processes. The os usually cleans up the mess but not within containers. I push the fix.