search

solo-io / solo-cop

Solo Communities of Practice
16 stars 9 forks source link

HA Performance Baseline/Testing #4

Open willowmck opened 2 years ago

willowmck commented 2 years ago

We should run routine HA baseline/performance testing. Need to decide on

This should be automate at a minimum weekly with a published report that can be shared internally.

willowmck commented 2 years ago

Performance based on length of reconcile loop

willowmck commented 2 years ago

@rvennam to contact @djannot on running perf benchmarks

rvennam commented 2 years ago 
Denis Jannot  4:51 AM
@Ram Vennam I did run the scale test with Gloo Mesh 1.2.3 (edited) 
10 clusters and 10000 Pods in total
30 minutes to discover all the workloads and destinations
1 minute to discover 10 new pods
After that, I have created 1000 VirtualDestinations (each one targeting 10 Pods, one per cluster) and 1000 TrafficPolicies
It took less than 10 minutes to have all of them ready on the Gloo Mesh side and another 15 minutes to have all the corresponding Istio objects (ServiceEntries, WorloadEntries, ...) created

All these numbers are not bad, but the main issue is when I modify a policy after everything is deployed. In that case it takes more than 10 minutes to process this change (edited)