search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.67k stars 4.75k forks source link

Crash at startup on 1.30.1 #35157

Closed nicoche closed 2 weeks ago

nicoche commented 1 month ago

Title: Crash at startup

Description:

We are seeing crashes at startup when using Envoy 1.30.1. This issue has been privately reported first, here is the initial discussion:

Hi,

Disclaimer: I don't think there is a security risk with this report, but Github's issue tracker form redirected me here to report a crash. Let me know if I should post over there instead.

We have random crashes when starting up Envoy.

$ envoy --version

envoy  version:
816188b86a0a52095b116b107f576324082c7c02/1.30.1/Distribution/RELEASE/BoringSSL

We have a XDS server delivering a conf. The crashes are seemingly random: around 50% of the time. The crashes happen on at least two platforms:

  • My laptop, M1 mac
  • CoreOS x86 servers

I attached a few backtraces of core dumps as I got them on my laptop. . Each time, I attached logs of Envoy. Logs of crash 4 have Envoy in debug mode (although it only captures the last 1000~ lines).

What I think is strange is that they don't all crash at the same spot. It looks (?) related to aggregate clusters.

Let me know if you need anything else to investigate, I'm happy to help.

Answer:

Thanks for the responsible report Nicolas. Looking at the information provided I think it would be safe to discuss this problem in the open. Please file an Issue on GitHub.

I'm unsure I have enough information to determine the root cause. These crashes have several CDS updates where some clusters were removed before the workers are started. It is possible it is a use-after-free in cluster manager. If you have ability to build Envoy with ASAN instrumentation, it could help in diagnosing the problem. Also do you know what clusters are removed during the CDS update?

Here are the backtraces I initially attached to the report:

Backtraces ``` * thread #8, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x0) * frame #0: 0x0000000106e98ad4 envoy`std::__1::__function::__func::makeSlotUpdateCb(std::__1::function)>)::'lambda'(std::__1::shared_ptr), std::__1::allocator::makeSlotUpdateCb(std::__1::function)>)::'lambda'(std::__1::shared_ptr)>, void (std::__1::shared_ptr)>::destroy_deallocate() + 56 frame #1: 0x0000000106e27524 envoy`std::__1::__function::__func)> const&)::$_1, std::__1::allocator)> const&)::$_1>, void ()>::destroy_deallocate() + 64 frame #2: 0x0000000105254938 envoy`void absl::lts_20230802::internal_any_invocable::RemoteManagerNontrivial>(absl::lts_20230802::internal_any_invocable::FunctionToCall, absl::lts_20230802::internal_any_invocable::TypeErasedState*, absl::lts_20230802::internal_any_invocable::TypeErasedState*) + 84 frame #3: 0x00000001075acd48 envoy`Envoy::Event::DispatcherImpl::runPostCallbacks() + 352 frame #4: 0x00000001075acbd4 envoy`Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) + 48 frame #5: 0x0000000106e4aa04 envoy`Envoy::Server::WorkerImpl::threadRoutine(Envoy::OptRef, std::__1::function const&) + 644 frame #6: 0x00000001078baf4c envoy`Envoy::Thread::PosixThreadFactoryImpl::createThread(std::__1::function, std::__1::optional const&, bool)::'lambda'(void*)::__invoke(void*) + 28 frame #7: 0x0000000183716034 libsystem_pthread.dylib`_pthread_start + 136 ``` ``` * thread #8, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x0) * frame #0: 0x000000010441fb5c envoy`Envoy::Extensions::Clusters::Aggregate::AggregateClusterLoadBalancer::onClusterAddOrUpdate(std::__1::basic_string_view>, std::__1::function&) + 488 frame #1: 0x0000000106795308 envoy`std::__1::__function::__func, void (Envoy::OptRef)>::operator()(Envoy::OptRef&&) + 2924 frame #2: 0x0000000106790bb8 envoy`decltype(std::declval::makeSlotUpdateCb(std::__1::function)>)::'lambda'(std::__1::shared_ptr)&>()(std::declval>())) std::__1::__invoke[abi:ue170006]::makeSlotUpdateCb(std::__1::function)>)::'lambda'(std::__1::shared_ptr)&, std::__1::shared_ptr>(Envoy::ThreadLocal::TypedSlot::makeSlotUpdateCb(std::__1::function)>)::'lambda'(std::__1::shared_ptr)&, std::__1::shared_ptr&&) + 76 frame #3: 0x000000010671f5e0 envoy`std::__1::__function::__func)> const&)::$_1, std::__1::allocator)> const&)::$_1>, void ()>::operator()() + 160 frame #4: 0x0000000106ea4d14 envoy`Envoy::Event::DispatcherImpl::runPostCallbacks() + 300 frame #5: 0x0000000106ea4bd4 envoy`Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) + 48 frame #6: 0x0000000106742a04 envoy`Envoy::Server::WorkerImpl::threadRoutine(Envoy::OptRef, std::__1::function const&) + 644 frame #7: 0x00000001071b2f4c envoy`Envoy::Thread::PosixThreadFactoryImpl::createThread(std::__1::function, std::__1::optional const&, bool)::'lambda'(void*)::__invoke(void*) + 28 frame #8: 0x0000000183716034 libsystem_pthread.dylib`_pthread_start + 136 ```

I'm following up in this Github issue with more details. First, I have more logs of a crash example.

The thread 6164983 crashes on cluster aggr_deployment_e911f492-2288-4c84-b03d-2a3804702c17 at 2024-07-11 16:47:12.407 ``` [2024-07-11 16:47:12.401][6164983][debug][main] [source/server/worker_impl.cc:143] worker entering dispatch loop [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster als_cluster [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster als_cluster added 1 removed 0 [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster xds_cluster [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster xds_cluster added 1 removed 0 [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster tracing:zipkin-collector [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster tracing:zipkin-collector added 1 removed 0 [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_e911f492-2288-4c84-b03d-2a3804702c17 [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_e911f492-2288-4c84-b03d-2a3804702c17' [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_72c5a459-e382-4bd1-8ba6-d720310f534b [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_72c5a459-e382-4bd1-8ba6-d720310f534b' [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_911a52c4-a8d4-4dcb-a4c6-0355e0f471dc [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_911a52c4-a8d4-4dcb-a4c6-0355e0f471dc' [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_cbd46ef0-266f-408b-8b1c-cd2aa1717ca0 [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_cbd46ef0-266f-408b-8b1c-cd2aa1717ca0' [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_aa751fba-9164-4b68-b368-0bcae184f8f7 [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_aa751fba-9164-4b68-b368-0bcae184f8f7' [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_c8275014-2810-4789-9141-55c2a33c02bd [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_c8275014-2810-4789-9141-55c2a33c02bd' [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_dde66d06-025a-4d67-9425-ababeff53bdb [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_dde66d06-025a-4d67-9425-ababeff53bdb' [2024-07-11 16:47:12.401][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_40c7cfa5-f4ae-4f88-a49a-a56d83264cc3 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_40c7cfa5-f4ae-4f88-a49a-a56d83264cc3' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_93c29cd2-c56f-4362-b011-184845cb9f7c [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_93c29cd2-c56f-4362-b011-184845cb9f7c' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_5fdd331d-2a01-4c98-8900-a4013daeffaf [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_5fdd331d-2a01-4c98-8900-a4013daeffaf' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_948e9aff-ffca-4ea4-bfc3-f7db1f3be823 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_948e9aff-ffca-4ea4-bfc3-f7db1f3be823' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_948e9aff-ffca-4ea4-bfc3-f7db1f3be823' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_c5d1dcc9-6165-4062-816d-f519146b1b8c [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_c5d1dcc9-6165-4062-816d-f519146b1b8c' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_dab45861-d0b8-4cf6-9ff3-d1869270a1ec [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_dab45861-d0b8-4cf6-9ff3-d1869270a1ec' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_f867320f-7358-4602-a8cc-6c39bdb9db57 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_f867320f-7358-4602-a8cc-6c39bdb9db57' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_3cf6d1ea-4046-497d-b4c0-d5c9edbf1625 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_3cf6d1ea-4046-497d-b4c0-d5c9edbf1625' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_d97c5e5a-e1c8-4dd4-a8f4-7fc8460b25e3 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_d97c5e5a-e1c8-4dd4-a8f4-7fc8460b25e3' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_9f8e20d7-3359-4159-827d-62d5b9c649f5 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_9f8e20d7-3359-4159-827d-62d5b9c649f5' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_ec7c552c-94f6-4cf4-8faf-b7462322c276 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_ec7c552c-94f6-4cf4-8faf-b7462322c276' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_c89ca89a-3aac-48a3-834e-a04f3dd70730 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_c89ca89a-3aac-48a3-834e-a04f3dd70730' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_1d2f65a4-5605-4cd1-9bd7-91c72dcb4ad7 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_1d2f65a4-5605-4cd1-9bd7-91c72dcb4ad7' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_f8027f84-ed4e-4322-ad03-cf03415a50cd [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_f8027f84-ed4e-4322-ad03-cf03415a50cd' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_2a6cd239-143d-4270-9b58-8da38e832e6c [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_2a6cd239-143d-4270-9b58-8da38e832e6c' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_2a6cd239-143d-4270-9b58-8da38e832e6c' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_50dd35d7-f5e7-418c-b378-7d0ce087418b [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_50dd35d7-f5e7-418c-b378-7d0ce087418b' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_8d2a2ff7-5e8d-48b8-bbc0-42158a090345 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_8d2a2ff7-5e8d-48b8-bbc0-42158a090345' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_4cfa419d-cae5-4a90-a238-2ef8daa2ee37 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_4cfa419d-cae5-4a90-a238-2ef8daa2ee37' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_a39aa0ca-1de7-4d69-aa31-8fb958e41f9c [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_a39aa0ca-1de7-4d69-aa31-8fb958e41f9c' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_a380d445-7998-45fc-9021-3db034f227e6 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_a380d445-7998-45fc-9021-3db034f227e6' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_a380d445-7998-45fc-9021-3db034f227e6' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_ca3824c4-145f-40b2-8c6c-1186aadf541c [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_ca3824c4-145f-40b2-8c6c-1186aadf541c' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_5fbcd280-4d79-49f3-bfd6-62355e41819f [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_5fbcd280-4d79-49f3-bfd6-62355e41819f' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_5fbcd280-4d79-49f3-bfd6-62355e41819f' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_651e0ae6-5668-4432-988e-7f599fc21e55 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_651e0ae6-5668-4432-988e-7f599fc21e55' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_44c842ca-a192-4c2a-ba00-73aebe3a813a [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_44c842ca-a192-4c2a-ba00-73aebe3a813a' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_857e8551-ec76-425e-805d-9143e8e41314 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_857e8551-ec76-425e-805d-9143e8e41314' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_857e8551-ec76-425e-805d-9143e8e41314' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_cba46b51-a537-43cd-b5aa-2df2c942b06c [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_cba46b51-a537-43cd-b5aa-2df2c942b06c' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_f6d7504d-9be1-4160-9472-1c9dddf9409e [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_f6d7504d-9be1-4160-9472-1c9dddf9409e' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_e06f333c-b584-43cb-8fe3-4fba693001b1 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_e06f333c-b584-43cb-8fe3-4fba693001b1' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_fee6283f-c5fc-4a74-88aa-e667652997c2 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_fee6283f-c5fc-4a74-88aa-e667652997c2' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_df741ac8-e251-4093-817a-b903a9869e87 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_df741ac8-e251-4093-817a-b903a9869e87' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_14ac5f3b-843e-4aa4-868c-c9fd28be5ee5 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_14ac5f3b-843e-4aa4-868c-c9fd28be5ee5' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_de2065e2-c045-48e2-8ebf-3bfa357144fb [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_de2065e2-c045-48e2-8ebf-3bfa357144fb' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_f4662d8a-59f8-4b2e-91b0-15e54d606cc9 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_f4662d8a-59f8-4b2e-91b0-15e54d606cc9' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_8e0cec10-02a7-4c1e-bcc9-f032f7d74592 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_8e0cec10-02a7-4c1e-bcc9-f032f7d74592' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_da931cbb-a1f8-40e9-af66-7181fdfb865b [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_da931cbb-a1f8-40e9-af66-7181fdfb865b' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_81cbf805-c053-4de8-949f-b090bd6bc3cf [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_81cbf805-c053-4de8-949f-b090bd6bc3cf' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_81cbf805-c053-4de8-949f-b090bd6bc3cf' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_c50be73b-fbd7-4223-9793-b9d436ccbfda [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_c50be73b-fbd7-4223-9793-b9d436ccbfda' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_f049b89d-203e-4bb8-826f-45f0726480ac [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_f049b89d-203e-4bb8-826f-45f0726480ac' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_f049b89d-203e-4bb8-826f-45f0726480ac' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_f049b89d-203e-4bb8-826f-45f0726480ac' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_693bb559-8dca-4483-ac4b-2ecb5fb80e4f [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_693bb559-8dca-4483-ac4b-2ecb5fb80e4f' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_5551d410-0820-4236-9b03-e12c285dbfe9 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_5551d410-0820-4236-9b03-e12c285dbfe9' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_3f29ddad-5887-4533-81a9-afd83ab1eb28 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_3f29ddad-5887-4533-81a9-afd83ab1eb28' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_043f21a9-b20c-48c1-ad94-14c813e4ef5a [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_043f21a9-b20c-48c1-ad94-14c813e4ef5a' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_d10d5cad-0e3f-4e7c-bfbb-e0d7b8ad7017 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_d10d5cad-0e3f-4e7c-bfbb-e0d7b8ad7017' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_ec5274d0-039b-4d01-9ca2-7e3833a3a952 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_ec5274d0-039b-4d01-9ca2-7e3833a3a952' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_ee9ed98a-549f-4724-a564-9ff680551176 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_ee9ed98a-549f-4724-a564-9ff680551176' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_9a80c70f-62f7-4f78-ba6e-c72fcb5fa534 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_9a80c70f-62f7-4f78-ba6e-c72fcb5fa534' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_0cbddbb0-81cb-46a0-9208-73ed24233291 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_0cbddbb0-81cb-46a0-9208-73ed24233291' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_9205607d-167e-40f9-a87b-e6bb26165988 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_9205607d-167e-40f9-a87b-e6bb26165988' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_4646cc7f-a4e4-4eb4-a473-0ab671f275cc [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_4646cc7f-a4e4-4eb4-a473-0ab671f275cc' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_21ae5a7b-4d59-475e-b6f2-d3f8e48f35e8 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_21ae5a7b-4d59-475e-b6f2-d3f8e48f35e8' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_70ee49ed-a8cc-4684-9e41-616b3315fb37 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_70ee49ed-a8cc-4684-9e41-616b3315fb37' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_d3ca1cda-e933-49a7-a7e1-d14610d52382 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_d3ca1cda-e933-49a7-a7e1-d14610d52382' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_58325e24-ac1c-4f9e-b0b2-18832b1dc675 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_58325e24-ac1c-4f9e-b0b2-18832b1dc675' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_202a09d9-c488-48c7-a2df-b106e5b2360c [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_202a09d9-c488-48c7-a2df-b106e5b2360c' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_3ebf01fc-ff54-4cf4-aa84-70f46b94a8f0 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_3ebf01fc-ff54-4cf4-aa84-70f46b94a8f0' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_97319a3e-0c77-4a73-8766-1279c91aa5df [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_97319a3e-0c77-4a73-8766-1279c91aa5df' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_b20fe586-3ab7-4cc1-b196-226401d134b2 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_b20fe586-3ab7-4cc1-b196-226401d134b2' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_deb22df8-a13b-4b1e-bffa-f11e00cb4022 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_deb22df8-a13b-4b1e-bffa-f11e00cb4022' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_2cea57c3-79c6-413a-9604-907c1c0c0072 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_2cea57c3-79c6-413a-9604-907c1c0c0072' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_a1e0e066-1a89-4ef2-957f-13b81d2bef82 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_a1e0e066-1a89-4ef2-957f-13b81d2bef82' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_f7bb5e69-77aa-4fe0-9086-eb6c36a68285 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_f7bb5e69-77aa-4fe0-9086-eb6c36a68285' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_3f5c3c7e-b33c-4035-aee5-6a9845b01405 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_3f5c3c7e-b33c-4035-aee5-6a9845b01405' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_9bbc64b2-6d75-4207-a6d8-2c31bdd02bbf [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_9bbc64b2-6d75-4207-a6d8-2c31bdd02bbf' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_fcac07c1-4a4e-4629-9f4a-edab1cb82403 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_fcac07c1-4a4e-4629-9f4a-edab1cb82403' [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_0318d649-2335-4eb3-90fa-f5cb6d3e0661 [2024-07-11 16:47:12.402][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_0318d649-2335-4eb3-90fa-f5cb6d3e0661' [2024-07-11 16:47:12.405][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_10e4303e-d91d-46af-81ce-5be549886d8c [2024-07-11 16:47:12.405][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_10e4303e-d91d-46af-81ce-5be549886d8c' [2024-07-11 16:47:12.405][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_f490d4dd-83b1-4a84-b053-5d3f771952c0 [2024-07-11 16:47:12.405][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_f490d4dd-83b1-4a84-b053-5d3f771952c0' [2024-07-11 16:47:12.405][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_8f04a20c-4e14-48c0-a33e-b5b1bbc20d59 [2024-07-11 16:47:12.405][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_8f04a20c-4e14-48c0-a33e-b5b1bbc20d59' [2024-07-11 16:47:12.405][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_62fdf90c-ff22-4e13-a6e6-7076534143a8 [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_62fdf90c-ff22-4e13-a6e6-7076534143a8' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_8a9d70e2-6734-47b2-9d2c-3857c7496aca [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_8a9d70e2-6734-47b2-9d2c-3857c7496aca' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_8cb12f23-455f-494c-934a-f7f161f79ed8 [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_8cb12f23-455f-494c-934a-f7f161f79ed8' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_86024d86-14eb-4349-8b1e-0126fa02e07b [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_86024d86-14eb-4349-8b1e-0126fa02e07b' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_885e0acb-c5b5-4c0a-b69a-dd78195a9839 [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_885e0acb-c5b5-4c0a-b69a-dd78195a9839' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_fd97dcef-6abf-4519-8e6b-d0ef0b36a0e5 [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_fd97dcef-6abf-4519-8e6b-d0ef0b36a0e5' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_7795da76-a1aa-4d57-b2d2-06bd5d8c4e81 [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_7795da76-a1aa-4d57-b2d2-06bd5d8c4e81' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_04b7f812-2329-4ab2-8197-4835a57c6800 [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_04b7f812-2329-4ab2-8197-4835a57c6800' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_9f150999-46a5-4e15-9d42-5bb9e7290b11 [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_9f150999-46a5-4e15-9d42-5bb9e7290b11' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_538cf731-606b-45b0-98c9-78c463197559 [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_538cf731-606b-45b0-98c9-78c463197559' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_f5ed516e-b637-47bc-9153-d4257b3b1426 [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_f5ed516e-b637-47bc-9153-d4257b3b1426' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_39e9df3f-8df2-4c1c-ba1b-41d17fea4a37 [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_39e9df3f-8df2-4c1c-ba1b-41d17fea4a37' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_9b349a9a-2188-4005-b487-ae4b3d9b1ff8 [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_9b349a9a-2188-4005-b487-ae4b3d9b1ff8' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_f49d7be1-2172-49a5-9d15-3291d6d6484a [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_f49d7be1-2172-49a5-9d15-3291d6d6484a' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_c748efd8-84e4-47a9-a693-a75ec69fb16d [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_c748efd8-84e4-47a9-a693-a75ec69fb16d' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_980a62f2-b498-4960-a05b-395442cafe99 [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_980a62f2-b498-4960-a05b-395442cafe99' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_87f27dcb-8df5-4e07-9330-e741a40661de [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_87f27dcb-8df5-4e07-9330-e741a40661de' [2024-07-11 16:47:12.406][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_73a87bbc-cc64-4fe4-8ec1-8cbacd389eae [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_73a87bbc-cc64-4fe4-8ec1-8cbacd389eae' [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_de86288b-b7c4-45ba-b154-fc4cf644e0bc [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_de86288b-b7c4-45ba-b154-fc4cf644e0bc' [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_b0b13254-922b-4b77-9350-cb31b492ce32 [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_b0b13254-922b-4b77-9350-cb31b492ce32' [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_893159e8-74e8-4589-bd16-7c529241776d [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kata' absent in aggregate cluster 'aggr_deployment_893159e8-74e8-4589-bd16-7c529241776d' [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_6cdc65fe-6aa2-41a7-8523-7dd7b6c2ab37 [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_6cdc65fe-6aa2-41a7-8523-7dd7b6c2ab37' [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_6cdc65fe-6aa2-41a7-8523-7dd7b6c2ab37' [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_c9233b1b-81aa-4189-98e5-d249376a69a7 [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_c9233b1b-81aa-4189-98e5-d249376a69a7' [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_465094c5-2eb8-4b0e-a40b-3b75ea3b5c8a [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_465094c5-2eb8-4b0e-a40b-3b75ea3b5c8a' [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_7ff9e27a-5362-495e-9f60-7e442a68d543 [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_7ff9e27a-5362-495e-9f60-7e442a68d543' [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_f382d26c-0b0c-4d6e-b02c-5c5e36c458a9 [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_f382d26c-0b0c-4d6e-b02c-5c5e36c458a9' [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_eb347d5b-4f96-461c-b938-f62cf75f93c9 [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_eb347d5b-4f96-461c-b938-f62cf75f93c9' [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_b548a048-a879-4cf5-b60e-bc92126e16a7 [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kata' absent in aggregate cluster 'aggr_deployment_b548a048-a879-4cf5-b60e-bc92126e16a7' [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d [2024-07-11 16:47:12.407][6164983][critical][backtrace] [./source/server/backtrace.h:127] Caught Segmentation fault: 11, suspect faulting address 0x0 [2024-07-11 16:47:12.407][6164983][critical][backtrace] [./source/server/backtrace.h:111] Backtrace (use tools/stack_decode.py to get line numbers): [2024-07-11 16:47:12.407][6164983][critical][backtrace] [./source/server/backtrace.h:112] Envoy version: 816188b86a0a52095b116b107f576324082c7c02/1.30.1/Distribution/RELEASE/BoringSSL [2024-07-11 16:47:12.407][6164983][critical][backtrace] [./source/server/backtrace.h:119] #0: _sigtramp [0x183745a24] ```
If we grep for this cluster name all logs, we can see that other threads are adding the same cluster at the same time as 6164983 ``` [2024-07-11 16:47:10.216][6164709][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:822] add/update cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d during init [2024-07-11 16:47:10.216][6164709][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:133] cm init: adding: cluster=aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d primary=0 secondary=6611 [2024-07-11 16:47:10.216][6164709][debug][upstream] [source/common/upstream/cds_api_helper.cc:51] cds: add/update cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:10.900][6164709][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:178] initializing secondary cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d [2024-07-11 16:47:10.900][6164709][debug][upstream] [source/common/upstream/upstream_impl.cc:1689] initializing Secondary cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d completed [2024-07-11 16:47:10.900][6164709][debug][init] [source/common/init/manager_impl.cc:49] init manager Cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d contains no targets [2024-07-11 16:47:10.900][6164709][debug][init] [source/common/init/watcher_impl.cc:14] init manager Cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d initialized, notifying ClusterImplBase [2024-07-11 16:47:10.900][6164709][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d [2024-07-11 16:47:10.900][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:10.900][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:10.900][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_tyo1-kuma' absent in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:10.900][6164709][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:167] cm init: init complete: cluster=aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d primary=0 secondary=8758 [2024-07-11 16:47:11.713][6164709][debug][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:93] gRPC config update took 3526 ms! Resources names: aggr_deployment_3bb1adc5-83be-4360-abb0- (long line, truncated) [2024-07-11 16:47:11.775][6164709][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:167] cm init: init complete: cluster=aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d primary=0 secondary=12 [2024-07-11 16:47:11.775][6164709][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:822] add/update cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d during init [2024-07-11 16:47:11.775][6164709][debug][upstream] [source/common/upstream/upstream_impl.cc:1689] initializing Secondary cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d completed [2024-07-11 16:47:11.775][6164709][debug][init] [source/common/init/manager_impl.cc:49] init manager Cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d contains no targets [2024-07-11 16:47:11.775][6164709][debug][init] [source/common/init/watcher_impl.cc:14] init manager Cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d initialized, notifying ClusterImplBase [2024-07-11 16:47:11.775][6164709][debug][init] [source/common/init/watcher_impl.cc:31] init manager Cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d destroyed [2024-07-11 16:47:11.775][6164709][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1283] updating TLS cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d [2024-07-11 16:47:11.775][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:11.775][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_tyo1-kuma' absent in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:11.775][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_fra1-kuma' absent in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:11.776][6164709][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:167] cm init: init complete: cluster=aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d primary=0 secondary=12 [2024-07-11 16:47:11.776][6164709][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:133] cm init: adding: cluster=aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d primary=0 secondary=12 [2024-07-11 16:47:11.776][6164709][debug][upstream] [source/common/upstream/cds_api_helper.cc:51] cds: add/update cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:11.822][6164709][debug][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:93] gRPC config update took 60 ms! Resources names: aggr_deployment_75e73201-cc95-4bdb-9c15-ceb (long line, truncated) [2024-07-11 16:47:11.859][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:120] adding or updating cluster 'dc_fra1-kuma' for aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:11.859][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_was1-kuma' absent in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:11.859][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_tyo1-kuma' absent in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:11.859][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:81] refresh: cluster 'dc_fra1-kuma' found in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:11.937][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:120] adding or updating cluster 'dc_was1-kuma' for aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:11.937][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:81] refresh: cluster 'dc_was1-kuma' found in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:11.937][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:77] refresh: cluster 'dc_tyo1-kuma' absent in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:11.937][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:81] refresh: cluster 'dc_fra1-kuma' found in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:12.272][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:120] adding or updating cluster 'dc_tyo1-kuma' for aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:12.272][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:81] refresh: cluster 'dc_was1-kuma' found in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:12.272][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:81] refresh: cluster 'dc_tyo1-kuma' found in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:12.272][6164709][debug][upstream] [source/extensions/clusters/aggregate/cluster.cc:81] refresh: cluster 'dc_fra1-kuma' found in aggregate cluster 'aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d' [2024-07-11 16:47:12.407][6164983][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d [2024-07-11 16:47:12.408][6164995][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d [2024-07-11 16:47:12.408][6164992][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1285] adding TLS cluster aggr_deployment_fd3310d5-c65f-4b40-ac93-0d27c717be9d ```

Then, to answer @yanavlasov questions:

If you have ability to build Envoy with ASAN instrumentation, it could help in diagnosing the problem.

I tried to, on Linux (./ci/run_envoy_docker.sh './ci/do_ci.sh asan') but failed miserably with:

ERROR: /source/test/common/buffer/BUILD:74:14: Linking test/common/buffer/buffer_util_test failed: (Exit 254): clang-14 failed: error executing command (from target //test/common/buffer:buffer_util_test) 
  (cd /build/.cache/envoy-bazel/bazel_root/base/sandbox/processwrapper-sandbox/10477/execroot/envoy && \
  exec env - \
    BAZEL_COMPILER=clang \
    BAZEL_CXXOPTS='-stdlib=libc++' \
    BAZEL_LINKLIBS=-l%:libc++.a:-l%:libc++abi.a \
    BAZEL_LINKOPTS=-lm:-pthread \
    CC=clang \
    CXX=clang++ \
    CXXFLAGS='-stdlib=libc++' \
    ENVOY_UBSAN_VPTR=1 \
    LDFLAGS='-stdlib=libc++' \
    LLVM_CONFIG=/opt/llvm/bin/llvm-config \
    PATH=/opt/llvm/bin:/opt/llvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
    PWD=/proc/self/cwd \
  /opt/llvm/bin/clang-14 @bazel-out/k8-dbg/bin/test/common/buffer/buffer_util_test-2.params)
# Configuration: 471f80c32d239c1a84b8c4dfea13399826edc2318edb798a1a409c57dd1fb3f8
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
clang-14: error: unable to execute command: Bus error
clang-14: error: linker command failed due to signal (use -v to see invocation)
[21,388 / 23,372] Compiling test/common/watchdog/abort_action_test.cc; 21s processwrapper-sandbox ... (2 actions running)
[21,390 / 23,372] 1 / 1208 tests, 1 failed; checking cached actions
INFO: Elapsed time: 45266.309s, Critical Path: 396.94s
INFO: 21390 processes: 10915 internal, 1 local, 10473 processwrapper-sandbox, 1 worker.
FAILED: Build did NOT complete successfully
//test/common/buffer:buffer_util_test                           FAILED TO BUILD

Executed 0 out of 1208 tests: 1 fails to build and 1207 were skipped.
/source/bazel-testlogs /source
cp: cannot stat '': No such file or directory

What did I do wrong?

Also do you know what clusters are removed during the CDS update?

It's a bit strange, I cannot tell. In the debug logs I pasted just before, there are mention of removed clusters:

(...)
[2024-07-11 16:47:08.186][6164709][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8863 cluster(s), remove 2 cluster(s)
[2024-07-11 16:47:10.165][6164709][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster tracing:zipkin-collector added 1 removed 0
[2024-07-11 16:47:11.765][6164709][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 8864 cluster(s), remove 2 cluster(s)
(...)

but this line is never matched: https://github.com/envoyproxy/envoy/blob/main/source/common/upstream/cluster_manager_impl.cc#L889

Any idea?

nezdolik commented 1 month ago

cc @yanavlasov (due to being involved in the ongoing conversation)

nicoche commented 1 month ago

Ok, I think that I got an easy repro.

Setup a XDS server. Write an infinite loop which:

At the end of the loop, sleep 3 seconds and begin again, but change the order of real clusters (e.g. ["real_cluster_b", "real_cluster_a", "real_cluster_c"])

Now, start a Envoy and make it connect to that XDS server. It will crash

github-actions[bot] commented 3 weeks ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions[bot] commented 2 weeks ago

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.