search

GoogleCloudPlatform / microservices-demo

Sample cloud-first application with 10 microservices showcasing Kubernetes, Istio, and gRPC.
https://cymbal-shops.retail.cymbal.dev
Apache License 2.0
16.18k stars 6.79k forks source link

Latency spikes observed in frontend #2400

Open dilinade opened 4 months ago

dilinade commented 4 months ago

I've deployed the microservices-demo on GKE, and while querying a single product (/product/{ID}) using Locust, I'm encountering intermittent latency spikes. The request rate is approximately 10 RPS. The spikes seem to originate from three different places

I enabled tracing and obtained the following traces:

Normal Trace : RTT 20ms Screenshot from 2024-02-29 09-49-57

Spike 1 : RTT 616ms Screenshot from 2024-02-29 09-48-59

Spike 2 : RTT 333ms Screenshot from 2024-02-29 09-49-17

Has anyone else encountered similar latency spikes? Are there any insights into why these spikes might occur, or is there any aspect that might have been overlooked in the deployment or configuration?

bourgeoisor commented 2 months ago

Hi @dilinade! I'm not able to reproduce this on my end. Are you still running into this?

dilinade commented 2 weeks ago

I believe the issue only arises when I set a CPU resource limit. I'm certain the usage doesn't exceed this limit; it's actually far below it, yet the problem still occurs.