Investigate intermittent response latency

f5devcentral / volt-demo-app

The Unlicense

6 stars 6 forks source link

Investigate intermittent response latency #16

Closed kreynoldsf5 closed 3 years ago

kreynoldsf5 commented 3 years ago

Intermittently, the response latency for a request will be disproportionately long (> 10s). Volterra dashboards suggests the latency is in the app (as opposed to ADN hops).

Investigate why this is happening.

Already reviewed container logs and didn't find anything
Already tweaked workload flavors but this doesn't appear to have any effect

kreynoldsf5 commented 3 years ago

Packet captured locally:

Not a DNS issue The delay happens after the TLS handshake.

We'll need to debug within the app.

kreynoldsf5 commented 3 years ago

Suspended the loadgen job. I was able to repro the issue after several requests.

This means it is not an 'ambient load' issue from loadgen (although the additional load may make the delay happen more often).

kreynoldsf5 commented 3 years ago

Including the adservice in the deployment.

This pushed resource usage above thresholds for some REs. I re-distributed the loadgen to the 'state' vsite (which now needs to be renamed).

Average response time is WAY lower. Longest single request in the past hour is 2.1 seconds (entering the ADN in EMEA).

I will continue to monitor but this looks promising.