Hot restarter kill sidecar immediately when current Envoy process is in INITIALIZING state

[biefy]

Bug Template

Title: Hot restarter kill sidecar immediately when current Envoy process is in INITIALIZING state

Description:

When hot_restarter.py receives HUP while the current envoy process is still having flags = INITIALIZING, hot_restarter.py simply exit with ABORT error (this is a silent death with nothing printed to stderr). This would kill Envoy sidecar container immediately. It would be ideal to provide an option telling hot_restarter.py to keep current Envoy process under this situation.

flags checking: https://github.com/envoyproxy/envoy/blob/master/source/server/hot_restart_impl.cc#L102

Repro steps:

Configure Envoy to a unreachable xDS server.
Use hot_restarter.py to start Envoy within a container. Then docker kill <container ID> --signal HUP

Config:

Static clusters + EDS.

static_resources:
listeners:
- name: listener_testdb
address:
socket_address: { address: 0.0.0.0, port_value: 53306 }
filter_chains:
- filters:
- name: envoy.tcp_proxy
config:
stat_prefix: testdb_egress
cluster: test_db_cluster
clusters:
- name: test_db_cluster
connect_timeout: 1s
type: EDS
lb_policy: ROUND_ROBIN
eds_cluster_config:
eds_config:
api_config_source:
api_type: GRPC
grpc_services:
envoy_grpc:
cluster_name: control_tower
- name: control_tower
connect_timeout: 1s
type: STRICT_DNS
http2_protocol_options: {}
load_assignment:
cluster_name: control_tower
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: xds.sample.com
port_value: 10080

Call Stack: I did not bother to build a debug binary. Instead, I used gdb to step through

[biefy]

@brian-pane This is the issue I mentioned earlier.

[zyfjeff]

@biefy Provide a retry option in this case, try again until you reach the largest number, what do you think?