matthoey-okta commented 5 years ago

When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.

Overview of the Issue

We recently discovered that when running consul maint or failing a health check, consul connect does not update envoy and removes the failed node. Turning on debug logs in envoy shows that the EDS from connect sends all hosts (healthy and unhealthy) instead of just healthy hosts. FWIW I'm only using 2 hosts in the upstream, but I don't think that should matter. Anyways, the hash is the same so envoy skips the updates. It's clear the watch is working because connect wants to send a new config, but the config is the same. Maybe some snapshot or caching error where we aren't sending the refreshed upstream?

Everything works with consul's proxy, but I noticed the logic in the code is different so that might explain why.

Reproduction Steps

Setup something with consul connect with envoy as the downstream. consul maint -enable on the upstream service and try to send data to it.

Consul info for both Client and Server

Client Info

``` agent: check_monitors = 0 check_ttls = 0 checks = 3 services = 2 build: prerelease = revision = c97c712e version = 1.4.2 consul: acl = disabled known_servers = 3 server = false runtime: arch = amd64 cpu_count = 2 goroutines = 75 max_procs = 2 os = linux version = go1.11.4 serf_lan: coordinate_resets = 0 encrypted = false event_queue = 0 event_time = 16 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 743 members = 8 query_queue = 0 query_time = 1 ```

Server info

``` agent: check_monitors = 0 check_ttls = 0 checks = 0 services = 0 build: prerelease = revision = c97c712e version = 1.4.2 consul: acl = disabled bootstrap = false known_datacenters = 1 leader = true leader_addr = 10.3.193.203:8300 server = true raft: applied_index = 174186159 commit_index = 174186159 fsm_pending = 0 last_contact = 0 last_log_index = 174186159 last_log_term = 29 last_snapshot_index = 174184567 last_snapshot_term = 29 latest_configuration = [{Suffrage:Voter ID:df1faae0-2457-e287-4926-9b92233caa70 Address:10.3.193.203:8300} {Suffrage:Voter ID:025c28c1-8731-dcc8-69bc-ceb718aa7d40 Address:10.3.198.160:8300} {Suffrage:Voter ID:17977d0f-9cea-42a5-9866-e75d762f9d6d Address:10.3.200.221:8300}] latest_configuration_index = 132243669 num_peers = 2 protocol_version = 3 protocol_version_max = 3 protocol_version_min = 0 snapshot_version_max = 1 snapshot_version_min = 0 state = Leader term = 29 runtime: arch = amd64 cpu_count = 2 goroutines = 160 max_procs = 2 os = linux version = go1.11.4 serf_lan: coordinate_resets = 0 encrypted = false event_queue = 0 event_time = 16 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 743 members = 8 query_queue = 0 query_time = 1 serf_wan: coordinate_resets = 0 encrypted = false event_queue = 0 event_time = 1 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 4 members = 1 query_queue = 0 query_time = 1 ```

Operating system and Environment details

amazon linux

Log Fragments

matthoey-okta commented 5 years ago

I noticed the code is slightly different for prepared queries so we changed our upstream to be a prepared query and the maint command now works properly with envoy. Hopefully that should help you all find the bug faster.

banks commented 5 years ago

@matthoey-okta thanks for the report! That does sound like a bug especially if Prepared Queries work as expected.

banks commented 5 years ago

@matthoey-okta I've dug into this and it's a little more subtle than I thought...

There is actually no difference I can see in the way Prepared Query and Services handle this case currently.

In both cases, we are reconfiguring the proxy to remove the failing instance(s) immediately. The problem you hit I think is to do with how Envoy handles updates where there are now zero healthy instances in a "cluster" (Envoy's idea of what we call an upstream).

We also have a bug now where we don't send responses when there are zero resources but even with that fixed (PR incoming) it doesn't actually change Envoy's behaviour here because Envoy refuses to forget about it's current known endpoints if you try to update it with a configuration that has zero healthy ones.

Turning on debug logs in envoy shows that the EDS from connect sends all hosts (healthy and unhealthy)

This is puzzling. Is it possible that you saw the initial delivery of the healthy endpoints before you put them in Maint mode? If you put them both in Maint mode while Envoy is running, the bug above means you'd never see another EDS update for them again until at least one of them came back. Would you be able to paste the Envoy debug output you see and describe the state of the cluster at the relevant times (e.g. when instances were OK and failing health)?

Also, you mentioned having two instances of the upstream, did you put both into Maint or just one?

I have a PR that fixes the issue I mentioned above that I'll open later and it may help with this because it also enables outlier detection which means that even if Envoy ignores an update that told it none of the instances were healthy, at least after they fail to connect a few times the metrics etc. will start showing them as unhealthy just because Envoy not considers them to be thanks to it's own outlier detection. But would be good to confirm exactly what you were seeing to be sure.

banks commented 5 years ago

After some more experimentation it turns out that Envoy does accept LoadAssignements with zero instances, I was just trying to be too clever and sending no resource at all in that case.

I think I can tweak my PR to make that work all as expected next week which should resolve this cleanly.

matthoey-okta commented 5 years ago

Thanks @banks . So you do think my test environment was correct (1 healthy, 1 unhealthy)?

Let me know as unfortunately I'm mid-migration to another test environment so might take me a second to reset everything, but could get it to you on Mon or Tues PT

banks commented 5 years ago

So you do think my test environment was correct (1 healthy, 1 unhealthy)?

Yes that should have worked as far as I understand it anyway (i.e. worked same for service or prepared query). I'll try to reproduce that again with that extra info. If you do manage to reproduce it would be useful to see Envoy debug logs and consul agent logs (the agent on same node as the misconfigured Envoy).

matthoey-okta commented 5 years ago

Consul on the source/downstream (referred to as client)

server-2_1          |     2019/03/19 00:18:45 [INFO] agent: Synced node info
server-2_1          |     2019/03/19 00:18:45 [INFO] serf: EventMemberJoin: http-server-adv-200-2 172.21.0.5
server-bootstrap_1  |     2019/03/19 00:18:45 [INFO] serf: EventMemberJoin: http-server-adv-200-2 172.21.0.5
server-bootstrap_1  |     2019/03/19 00:18:45 [INFO] consul: member 'http-server-adv-200-2' joined, marking health alive
server-bootstrap_1  |     2019/03/19 00:18:45 [INFO] serf: EventMemberJoin: http-server-adv-200-1 172.21.0.6
server-bootstrap_1  |     2019/03/19 00:18:45 [INFO] consul: member 'http-server-adv-200-1' joined, marking health alive
server-1_1          |     2019/03/19 00:18:45 [INFO] serf: EventMemberJoin: http-server-adv-200-2 172.21.0.5
server-1_1          |     2019/03/19 00:18:45 [INFO] serf: EventMemberJoin: http-server-adv-200-1 172.21.0.6
server-2_1          |     2019/03/19 00:18:45 [INFO] serf: EventMemberJoin: http-server-adv-200-1 172.21.0.6
server-1_1          | 2019/03/19 00:18:46 [DEBUG] raft-net: 172.21.0.4:8300 accepted connection from: 172.21.0.2:45532
server-bootstrap_1  | ==> Newer Consul version available: 1.4.3 (currently running: 1.4.2)
server-bootstrap_1  |     2019/03/19 00:18:58 [INFO] serf: EventMemberJoin: http-client-adv-1 172.21.0.7
server-bootstrap_1  |     2019/03/19 00:18:58 [INFO] consul: member 'http-client-adv-1' joined, marking health alive
server-2_1          |     2019/03/19 00:18:58 [INFO] serf: EventMemberJoin: http-client-adv-1 172.21.0.7
server-1_1          |     2019/03/19 00:18:58 [INFO] serf: EventMemberJoin: http-client-adv-1 172.21.0.7
server-2_1          | ==> Newer Consul version available: 1.4.3 (currently running: 1.4.2)
server-2_1          |     2019/03/19 00:20:44 [INFO] agent: Node entered maintenance mode
server-2_1          |     2019/03/19 00:20:44 [INFO] agent: Synced check "_node_maintenance"
server-2_1          |     2019/03/19 00:24:02 [INFO] agent: Node left maintenance mode
server-2_1          |     2019/03/19 00:24:02 [INFO] agent: Deregistered check "_node_maintenance"
server-2_1          |     2019/03/19 00:26:01 [INFO] memberlist: Suspect http-client-adv-1 has failed, no acks received
server-bootstrap_1  |     2019/03/19 00:26:03 [INFO] memberlist: Suspect http-client-adv-1 has failed, no acks received
server-bootstrap_1  |     2019/03/19 00:26:04 [INFO] memberlist: Marking http-client-adv-1 as failed, suspect timeout reached (2 peer confirmations)
server-bootstrap_1  |     2019/03/19 00:26:04 [INFO] serf: EventMemberFailed: http-client-adv-1 172.21.0.7
server-bootstrap_1  |     2019/03/19 00:26:04 [INFO] consul: member 'http-client-adv-1' failed, marking health critical
server-1_1          |     2019/03/19 00:26:04 [INFO] memberlist: Marking http-client-adv-1 as failed, suspect timeout reached (2 peer confirmations)
server-1_1          |     2019/03/19 00:26:04 [INFO] serf: EventMemberFailed: http-client-adv-1 172.21.0.7
server-2_1          |     2019/03/19 00:26:04 [INFO] memberlist: Marking http-client-adv-1 as failed, suspect timeout reached (2 peer confirmations)
server-2_1          |     2019/03/19 00:26:04 [INFO] serf: EventMemberFailed: http-client-adv-1 172.21.0.7
server-bootstrap_1  |     2019/03/19 00:26:07 [INFO] memberlist: Suspect http-server-adv-200-1 has failed, no acks received
server-1_1          |     2019/03/19 00:26:08 [INFO] memberlist: Suspect http-server-adv-200-2 has failed, no acks received

Envoy

[2019-03-19 00:19:59.849][000054][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:19:59.849][000054][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:19:59.849][000054][debug][main] [source/server/connection_handler_impl.cc:68] [C43] adding to cleanup list
[2019-03-19 00:19:59.849][000054][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:19:59.849][000054][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:01.848][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:20:02.465][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C44] new connection
[2019-03-19 00:20:02.571][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:02.572][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:02.572][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:02.572][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C44] adding to cleanup list
[2019-03-19 00:20:02.572][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:02.572][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:03.372][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:03.372][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C46] adding to cleanup list
[2019-03-19 00:20:03.372][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:03.372][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:03.463][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:03.463][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:06.579][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C48] new connection
[2019-03-19 00:20:06.584][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:06.584][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:06.584][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:06.584][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C48] adding to cleanup list
[2019-03-19 00:20:06.584][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:06.584][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:06.866][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:20:07.576][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:07.576][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:10.696][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:10.696][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:10.696][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:10.697][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C50] adding to cleanup list
[2019-03-19 00:20:10.697][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:10.697][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:11.589][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:11.870][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:20:13.374][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C52] new connection
[2019-03-19 00:20:13.375][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:13.375][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C52] adding to cleanup list
[2019-03-19 00:20:13.375][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:13.375][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:14.703][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C54] new connection
[2019-03-19 00:20:14.807][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:14.807][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:14.807][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:14.808][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C54] adding to cleanup list
[2019-03-19 00:20:14.808][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:14.808][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:15.699][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:15.699][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:16.877][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:20:18.816][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C56] new connection
[2019-03-19 00:20:18.820][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:18.820][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:18.821][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:18.821][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C56] adding to cleanup list
[2019-03-19 00:20:18.821][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:18.821][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:19.812][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:19.812][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:21.889][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:20:22.827][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C58] new connection
[2019-03-19 00:20:22.934][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:22.934][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:22.934][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:22.935][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:22.935][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:23.376][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C60] new connection
[2019-03-19 00:20:23.376][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:23.377][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C60] adding to cleanup list
[2019-03-19 00:20:23.377][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:23.377][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:23.823][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:23.823][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:26.875][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:20:26.918][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C62] new connection
[2019-03-19 00:20:26.924][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:26.924][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:26.924][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:26.924][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C62] adding to cleanup list
[2019-03-19 00:20:26.924][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:26.924][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:27.916][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:27.916][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:30.931][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C64] new connection
[2019-03-19 00:20:30.934][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:30.934][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:30.934][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:30.935][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C64] adding to cleanup list
[2019-03-19 00:20:30.935][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:30.935][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:31.876][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:20:31.924][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:31.924][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:33.357][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C66] new connection
[2019-03-19 00:20:33.357][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:33.358][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C66] adding to cleanup list
[2019-03-19 00:20:33.358][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:33.358][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:34.941][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C68] new connection
[2019-03-19 00:20:35.046][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:35.046][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:35.046][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:35.047][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:35.047][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:35.935][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:35.935][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:36.882][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:20:39.052][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C70] new connection
[2019-03-19 00:20:39.156][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:39.156][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:39.156][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:39.157][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C70] adding to cleanup list
[2019-03-19 00:20:39.157][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:39.157][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:40.046][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:40.046][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:41.886][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:20:43.163][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C72] new connection
[2019-03-19 00:20:43.167][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:43.167][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:43.167][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:43.167][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C72] adding to cleanup list
[2019-03-19 00:20:43.167][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:43.167][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:43.359][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C74] new connection
[2019-03-19 00:20:43.359][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:43.359][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C74] adding to cleanup list
[2019-03-19 00:20:43.359][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:43.359][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:44.159][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:44.159][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:46.888][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:20:47.174][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C76] new connection
[2019-03-19 00:20:47.278][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:47.279][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:47.279][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:47.279][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C76] adding to cleanup list
[2019-03-19 00:20:47.279][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:47.279][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:48.170][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:48.171][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:51.286][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C78] new connection
[2019-03-19 00:20:51.291][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:51.291][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:51.291][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C78] adding to cleanup list
[2019-03-19 00:20:51.291][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:51.291][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:51.889][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:20:52.281][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:52.281][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:53.360][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C80] new connection
[2019-03-19 00:20:53.360][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:53.361][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C80] adding to cleanup list
[2019-03-19 00:20:53.361][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:53.361][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:55.297][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C82] new connection
[2019-03-19 00:20:55.403][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:55.403][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:55.403][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:55.404][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:55.404][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:56.270][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:56.270][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:20:56.862][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:20:59.389][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C84] new connection
[2019-03-19 00:20:59.395][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:59.395][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:20:59.395][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:20:59.396][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C84] adding to cleanup list
[2019-03-19 00:20:59.396][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:20:59.396][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:00.386][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:00.386][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:01.865][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:21:03.339][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C86] new connection
[2019-03-19 00:21:03.339][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:03.339][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C86] adding to cleanup list
[2019-03-19 00:21:03.339][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:03.339][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:03.402][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C88] new connection
[2019-03-19 00:21:03.405][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:03.405][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:03.405][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:03.405][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:03.405][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:04.397][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:04.398][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:06.864][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:21:07.413][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C90] new connection
[2019-03-19 00:21:07.416][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:07.416][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:07.417][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:07.420][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C90] adding to cleanup list
[2019-03-19 00:21:07.421][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:07.421][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:08.407][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:08.407][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:11.426][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C92] new connection
[2019-03-19 00:21:11.532][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:11.532][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:11.532][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:11.536][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C92] adding to cleanup list
[2019-03-19 00:21:11.536][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:11.536][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:11.864][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:21:12.417][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:12.417][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:13.340][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C94] new connection
[2019-03-19 00:21:13.340][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:13.340][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C94] adding to cleanup list
[2019-03-19 00:21:13.340][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:13.340][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:15.543][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C96] new connection
[2019-03-19 00:21:15.646][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:15.646][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:15.647][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:15.647][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:15.647][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:16.533][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:16.533][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:16.870][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:21:19.653][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C98] new connection
[2019-03-19 00:21:19.757][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:19.757][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:19.757][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:19.757][000070][debug][main] [source/server/connection_handler_impl.cc:68] [C98] adding to cleanup list
[2019-03-19 00:21:19.757][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:19.757][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:20.648][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:20.648][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:21.869][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:21:23.342][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C100] new connection
[2019-03-19 00:21:23.342][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:23.342][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C100] adding to cleanup list
[2019-03-19 00:21:23.342][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:23.342][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:23.764][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C102] new connection
[2019-03-19 00:21:23.768][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:23.768][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:23.768][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:23.768][000070][debug][main] [source/server/connection_handler_impl.cc:68] [C102] adding to cleanup list
[2019-03-19 00:21:23.768][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:23.768][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:24.759][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:26.846][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:21:27.859][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:27.859][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:27.860][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C104] adding to cleanup list
[2019-03-19 00:21:27.860][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:27.860][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:28.749][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:28.749][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:31.850][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:21:31.864][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C106] new connection
[2019-03-19 00:21:31.869][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:31.869][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:31.869][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:31.871][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C106] adding to cleanup list
[2019-03-19 00:21:31.871][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:31.871][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:32.862][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:32.862][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:33.321][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C108] new connection
[2019-03-19 00:21:33.321][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C108] adding to cleanup list
[2019-03-19 00:21:33.321][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:33.321][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:33.321][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:35.875][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C110] new connection
[2019-03-19 00:21:35.878][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:35.878][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:35.878][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C110] adding to cleanup list
[2019-03-19 00:21:35.879][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:35.879][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:36.859][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:21:36.870][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:36.870][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:39.888][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:39.888][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:39.888][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:39.890][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C112] adding to cleanup list
[2019-03-19 00:21:39.890][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:39.890][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:40.880][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:40.880][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:41.863][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:21:43.323][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C114] new connection
[2019-03-19 00:21:43.323][000070][debug][main] [source/server/connection_handler_impl.cc:68] [C114] adding to cleanup list
[2019-03-19 00:21:43.323][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:43.323][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:43.323][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:44.002][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:44.002][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:44.002][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:44.003][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C116] adding to cleanup list
[2019-03-19 00:21:44.003][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:44.003][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:44.890][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:46.867][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:21:48.013][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:48.013][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:48.013][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:48.016][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C118] adding to cleanup list
[2019-03-19 00:21:48.016][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:48.016][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:49.006][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:49.006][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:51.865][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:21:52.022][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C120] new connection
[2019-03-19 00:21:52.126][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:52.126][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:52.126][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:52.127][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C120] adding to cleanup list
[2019-03-19 00:21:52.127][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:52.127][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:53.014][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:53.015][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:53.325][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C122] new connection
[2019-03-19 00:21:53.326][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:53.326][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C122] adding to cleanup list
[2019-03-19 00:21:53.326][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:53.326][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:56.111][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C124] new connection
[2019-03-19 00:21:56.215][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:21:56.215][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:21:56.215][000070][debug][main] [source/server/connection_handler_impl.cc:68] [C124] adding to cleanup list
[2019-03-19 00:21:56.215][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:56.215][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:21:56.858][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:21:57.107][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:21:57.107][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:00.325][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:00.325][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:00.325][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:00.326][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C126] adding to cleanup list
[2019-03-19 00:22:00.326][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:00.326][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:01.217][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:01.217][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:01.865][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:22:03.305][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C128] new connection
[2019-03-19 00:22:03.306][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:03.306][000070][debug][main] [source/server/connection_handler_impl.cc:68] [C128] adding to cleanup list
[2019-03-19 00:22:03.306][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:03.306][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:04.337][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:04.337][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:04.337][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:04.337][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:04.337][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:05.328][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:05.328][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:06.875][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:22:08.344][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C132] new connection
[2019-03-19 00:22:08.449][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:08.450][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:08.450][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:08.452][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:08.452][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:09.341][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:09.341][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:11.879][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:22:12.459][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C134] new connection
[2019-03-19 00:22:12.464][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:12.464][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:12.464][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:12.464][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:12.464][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:13.309][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C136] new connection
[2019-03-19 00:22:13.309][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:13.309][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C136] adding to cleanup list
[2019-03-19 00:22:13.309][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:13.309][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:13.453][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:13.453][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:16.473][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:16.473][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:16.473][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:16.474][000070][debug][main] [source/server/connection_handler_impl.cc:68] [C138] adding to cleanup list
[2019-03-19 00:22:16.474][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:16.474][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:16.887][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:22:17.465][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:17.465][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:20.585][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:20.585][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:20.585][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:20.586][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C140] adding to cleanup list
[2019-03-19 00:22:20.586][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:20.586][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:21.476][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:21.476][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:21.890][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:22:23.311][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C142] new connection
[2019-03-19 00:22:23.311][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:23.311][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C142] adding to cleanup list
[2019-03-19 00:22:23.311][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:23.311][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:24.592][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C144] new connection
[2019-03-19 00:22:24.696][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:24.696][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:24.696][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:24.696][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C144] adding to cleanup list
[2019-03-19 00:22:24.696][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:24.696][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:25.566][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:26.869][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:22:28.687][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:28.687][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:28.687][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:28.688][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C146] adding to cleanup list
[2019-03-19 00:22:28.688][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:28.688][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:29.679][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:31.878][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:22:32.699][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:32.699][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:32.699][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:32.702][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C148] adding to cleanup list
[2019-03-19 00:22:32.702][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:32.702][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:33.290][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C150] new connection
[2019-03-19 00:22:33.290][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:33.291][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C150] adding to cleanup list
[2019-03-19 00:22:33.291][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:33.291][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:33.689][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:36.709][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C152] new connection
[2019-03-19 00:22:36.815][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:36.815][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:36.815][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:36.816][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C152] adding to cleanup list
[2019-03-19 00:22:36.816][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:36.816][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:37.700][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:37.700][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:40.825][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:40.825][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:40.825][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:40.826][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C154] adding to cleanup list
[2019-03-19 00:22:40.826][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:40.826][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:41.817][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:41.886][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:22:43.291][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C156] new connection
[2019-03-19 00:22:43.291][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:43.291][000070][debug][main] [source/server/connection_handler_impl.cc:68] [C156] adding to cleanup list
[2019-03-19 00:22:43.291][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:43.291][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:44.939][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:44.939][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:44.939][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:44.940][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:44.940][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:45.827][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:45.827][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:46.886][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:22:48.946][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C160] new connection
[2019-03-19 00:22:49.050][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:49.050][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:49.050][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:49.051][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:49.051][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:49.939][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:49.939][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:51.889][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:22:53.058][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C162] new connection
[2019-03-19 00:22:53.063][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:53.063][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:53.063][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:53.066][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C162] adding to cleanup list
[2019-03-19 00:22:53.067][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:53.067][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:53.293][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C164] new connection
[2019-03-19 00:22:53.293][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:53.293][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C164] adding to cleanup list
[2019-03-19 00:22:53.293][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:53.293][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:54.051][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:54.051][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:56.873][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:22:57.051][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C166] new connection
[2019-03-19 00:22:57.157][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:57.157][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:22:57.157][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:22:57.158][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C166] adding to cleanup list
[2019-03-19 00:22:57.158][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:22:57.158][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:22:58.044][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:01.163][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C168] new connection
[2019-03-19 00:23:01.270][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:01.270][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:01.270][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:01.271][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:01.271][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:01.872][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:23:02.158][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:02.158][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:03.274][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C170] new connection
[2019-03-19 00:23:03.274][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:03.274][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C170] adding to cleanup list
[2019-03-19 00:23:03.274][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:03.274][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:05.278][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C172] new connection
[2019-03-19 00:23:05.283][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:05.283][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:05.283][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:05.283][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C172] adding to cleanup list
[2019-03-19 00:23:05.284][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:05.284][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:06.270][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:06.270][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:06.869][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:23:09.291][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C174] new connection
[2019-03-19 00:23:09.294][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:09.294][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:09.294][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:09.294][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:09.294][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:10.283][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:10.283][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:11.869][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:23:13.275][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C176] new connection
[2019-03-19 00:23:13.275][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:13.275][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C176] adding to cleanup list
[2019-03-19 00:23:13.275][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:13.275][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:13.302][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C178] new connection
[2019-03-19 00:23:13.407][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:13.407][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:13.408][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:13.408][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C178] adding to cleanup list
[2019-03-19 00:23:13.408][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:13.408][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:14.297][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:14.297][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:16.871][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:23:17.413][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C180] new connection
[2019-03-19 00:23:17.517][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:17.517][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:17.517][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:17.517][000070][debug][main] [source/server/connection_handler_impl.cc:68] [C180] adding to cleanup list
[2019-03-19 00:23:17.517][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:17.517][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:18.410][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:21.523][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C182] new connection
[2019-03-19 00:23:21.527][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:21.527][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:21.527][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:21.528][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C182] adding to cleanup list
[2019-03-19 00:23:21.528][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:21.528][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:22.519][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:22.519][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:23.277][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C184] new connection
[2019-03-19 00:23:23.278][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C184] adding to cleanup list
[2019-03-19 00:23:23.278][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:23.278][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:23.278][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:25.511][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C186] new connection
[2019-03-19 00:23:25.514][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:25.514][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:25.514][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:25.515][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C186] adding to cleanup list
[2019-03-19 00:23:25.515][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:25.515][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:26.507][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:26.507][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:26.855][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:23:29.520][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C188] new connection
[2019-03-19 00:23:29.629][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:29.629][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:29.629][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:29.629][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C188] adding to cleanup list
[2019-03-19 00:23:29.629][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:29.629][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:30.520][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:30.520][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:31.866][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:23:33.258][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C190] new connection
[2019-03-19 00:23:33.258][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:33.258][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C190] adding to cleanup list
[2019-03-19 00:23:33.258][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:33.258][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:33.635][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C192] new connection
[2019-03-19 00:23:33.739][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:33.739][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:33.739][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:33.739][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:33.739][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:34.632][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:34.632][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:36.884][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:23:37.744][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C194] new connection
[2019-03-19 00:23:37.749][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:37.749][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:37.749][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:37.749][000070][debug][main] [source/server/connection_handler_impl.cc:68] [C194] adding to cleanup list
[2019-03-19 00:23:37.749][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:37.749][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:38.743][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:41.758][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:41.758][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:41.758][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:41.759][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:41.759][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:41.887][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:23:42.750][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:42.750][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:43.259][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C198] new connection
[2019-03-19 00:23:43.260][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:43.260][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C198] adding to cleanup list
[2019-03-19 00:23:43.260][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:43.261][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:46.759][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:46.759][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:23:51.891][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:23:53.260][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C200] new connection
[2019-03-19 00:23:53.261][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:23:53.261][000070][debug][main] [source/server/connection_handler_impl.cc:68] [C200] adding to cleanup list
[2019-03-19 00:23:53.261][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:23:53.261][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:23:56.881][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:24:01.889][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:24:03.239][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C202] new connection
[2019-03-19 00:24:03.240][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:24:03.240][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C202] adding to cleanup list
[2019-03-19 00:24:03.240][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:24:03.240][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:24:06.894][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:24:11.904][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:24:13.240][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C204] new connection
[2019-03-19 00:24:13.241][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:24:13.241][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C204] adding to cleanup list
[2019-03-19 00:24:13.241][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:24:13.241][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:24:16.908][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:24:21.908][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:24:23.242][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C206] new connection
[2019-03-19 00:24:23.242][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C206] adding to cleanup list
[2019-03-19 00:24:23.242][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:24:23.242][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:24:23.242][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:24:26.887][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:24:31.892][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:24:33.222][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C208] new connection
[2019-03-19 00:24:33.222][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:24:33.222][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C208] adding to cleanup list
[2019-03-19 00:24:33.222][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:24:33.222][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:24:36.897][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:24:41.893][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:24:43.226][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C210] new connection
[2019-03-19 00:24:43.226][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C210] adding to cleanup list
[2019-03-19 00:24:43.226][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:24:43.226][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:24:43.226][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:24:46.896][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:24:51.893][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:24:52.459][000054][debug][main] [source/server/connection_handler_impl.cc:257] [C212] new connection
[2019-03-19 00:24:52.459][000054][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:24:52.459][000054][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:24:52.460][000054][debug][main] [source/server/connection_handler_impl.cc:68] [C212] adding to cleanup list
[2019-03-19 00:24:52.460][000054][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:24:52.460][000054][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=1)
[2019-03-19 00:24:53.228][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C213] new connection
[2019-03-19 00:24:53.228][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:24:53.228][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C213] adding to cleanup list
[2019-03-19 00:24:53.228][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:24:53.228][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:24:56.871][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:25:01.875][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:25:03.208][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C215] new connection
[2019-03-19 00:25:03.208][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:25:03.208][000070][debug][main] [source/server/connection_handler_impl.cc:68] [C215] adding to cleanup list
[2019-03-19 00:25:03.208][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:25:03.208][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:25:06.873][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:25:11.873][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:25:13.210][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C217] new connection
[2019-03-19 00:25:13.210][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C217] adding to cleanup list
[2019-03-19 00:25:13.210][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:25:13.210][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:25:13.210][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:25:16.872][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:25:21.878][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:25:23.210][000073][debug][main] [source/server/connection_handler_impl.cc:257] [C219] new connection
[2019-03-19 00:25:23.211][000073][debug][main] [source/server/connection_handler_impl.cc:68] [C219] adding to cleanup list
[2019-03-19 00:25:23.211][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:25:23.211][000073][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:25:23.211][000073][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:25:26.853][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:25:31.852][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:25:33.190][000072][debug][main] [source/server/connection_handler_impl.cc:257] [C221] new connection
[2019-03-19 00:25:33.190][000072][debug][main] [source/server/connection_handler_impl.cc:68] [C221] adding to cleanup list
[2019-03-19 00:25:33.190][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:25:33.190][000072][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:25:33.190][000072][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:25:36.854][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:25:41.853][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:25:43.191][000071][debug][main] [source/server/connection_handler_impl.cc:257] [C223] new connection
[2019-03-19 00:25:43.192][000071][debug][main] [source/server/connection_handler_impl.cc:68] [C223] adding to cleanup list
[2019-03-19 00:25:43.192][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:25:43.192][000071][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:25:43.192][000071][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:25:46.849][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:25:51.857][000054][debug][main] [source/server/server.cc:143] flushing stats
[2019-03-19 00:25:53.193][000070][debug][main] [source/server/connection_handler_impl.cc:257] [C225] new connection
[2019-03-19 00:25:53.193][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=1)
[2019-03-19 00:25:53.193][000070][debug][main] [source/server/connection_handler_impl.cc:68] [C225] adding to cleanup list
[2019-03-19 00:25:53.193][000070][trace][main] [source/common/event/dispatcher_impl.cc:130] item added to deferred deletion list (size=2)
[2019-03-19 00:25:53.193][000070][trace][main] [source/common/event/dispatcher_impl.cc:56] clearing deferred deletion list (size=2)
[2019-03-19 00:25:56.841][000054][debug][main] [source/server/server.cc:143] flushing stats

matthoey-okta commented 5 years ago

One other thing we're trying to figure out that is different in two environments, but here is the envoy messages I originally saw. The only difference between the two environments is the network being host based vs bridged in docker. This pops up every time I maintenance a host on our other environment. Again, two hosts healthy, one is maint'd and then disabled:

 [cluster_name: "service:test-monoapp"
endpoints {
  lb_endpoints {
    endpoint {
      address {
        socket_address {
          address: "10.3.193.128"
          port_value: 9005
        }
      }
    }
  }
  lb_endpoints {
    endpoint {
      address {
        socket_address {
          address: "10.3.198.209"
          port_value: 9005
        }
      }
    }
  }
}

banks commented 5 years ago

@matthoey-okta Thanks for the details. The logs I think confirms the issue - Envoy didn't see any LoadAssignment updates at all (not even empty ones) like the one you pasted in your last message during the period where the maint check is added and released.

I'm now pretty sure that is because of the issue I dug up while looking into this: #5506 which I just pushed a PR to fix (#5508) and not due to the empty set of nodes issue I was talking about earlier (which is also fixed in #5499 anyway).

The tl;dr is that our Connect service discovery has always had a bug due to the subtleties with which we manage blocking query indexes for services that should be fixed with that new PR.

Thanks again for your help tracking this down! I expect the fixes to be in a release within a few weeks.

matthoey-okta commented 5 years ago

@banks Finally got around to testing your fix in #5508 (actually just built off of 9d6e5076164e9c819af250db8fdfcf235e5ed4b6 in master today,) and confirmed things now work. Awesome!

Wonder if I found a different issue though. Normally we were adding a maint to the entire host, however today I finally got around to setting up an environment to reproduce (https://gist.github.com/matthoey-okta/6ef2208616a4b1f2202d55886aa27e51 has instructions) and noticed that adding a maint to the base service (e.g. socat9000) does not prevent envoy traffic from flowing. You must also maint the sidecar-proxy service as well. Should aliasing handle this?

banks commented 5 years ago

You must also maint the sidecar-proxy service as well. Should aliasing handle this?

Yes that would be a bug.

But... can you check something for me? In the repro gist you posted the socat9000 service is being added with explicit check:

"check": {
  "id": "tcp9000",
  "name": "tcp 9000 check",
  "tcp": "localhost:9000",
  "interval": "10s",
  "timeout": "5s"
},

Per the docs, if you override check/checks in the sidecar_service, it overrides all of them and since there is no Alias check defined above the socat9000 service proxy actually won't have an alias check at all.

Can you check if the registered proxy service has an alias check defined? I expect not which explains the behaviour you're seeing!

If you remove the above sidecar_service check definition then the default will be to add both a TCP check for the public listener AND the alias check. If you do wish to override the default TCP check for some reason then you can also add the alias check explicitly and all should work again.

Glad the original issue is now resolved.

Thanks!

I'll close this for now but feel free to add more info if you still see issues related to this!

matthoey-okta commented 5 years ago

@banks Yeah, it has a check called "Connect Sidecar Aliasing socat9000". socat is actually off right now so it currently says Aliased check "tcp 9000 check" failing:

Though I'm a bit confused by your comment. I'm not overriding checks for the sidecar_service am I? That check is on the main service and then I just create the sidecar_service using the short hand "connect": { "sidecar_service": {} } which should automatically create the alias check.

However, that's likely why the maint issue would exist. It probably only has aliases for the checks that are defined when the agent reloads. maint will create a new check and set it to critical. Unless there's special code in the maint to examine if it's a connect enabled service and then find the service which has the destination_service_id option set to the service you're enabling maint on and maint it as well I could see why this issue would exist.

banks commented 5 years ago

@matthoey-okta Thanks, you are right - I misread that config. The TCP check is on the service not the sidecar.

So this does seem to be a real issue still.

Alias checks literally setup a watch on the target service and flip their status any time the target service's status changes. For the common case when they are on the same node they don't do that via the API for performance reasons but internally in the agent's memory.

So any check that makes a service critical should also affect a sidecar with an alias check for that service. If that isn't the case then it does seem like a real bug. Off the top of my head, I can't think of a reason that maint checks are any different from normal ones in a way that would explain why they don't work when others do.

banks commented 5 years ago

Actually I'll open a new issue for this to get a clearer problem statement.

hashicorp / consul

Consul Connect Doesn't Update Envoy with Consul Maint or Failed Healthchecks #5332

Overview of the Issue

Reproduction Steps

Consul info for both Client and Server

Operating system and Environment details

Log Fragments

Consul on the source/downstream (referred to as client)

Envoy