search

apache / trafficserver

Apache Traffic Server™ is a fast, scalable and extensible HTTP/1.1 and HTTP/2 compliant caching proxy server.
https://trafficserver.apache.org/
Apache License 2.0
1.82k stars 804 forks source link

ATS7,8,9 remap_stats: wrong stat ids assigned after hard traffic_server crash #6237

Open traeak opened 4 years ago

traeak commented 4 years ago

Recreated with 0cf7e5953aec06d4dafb183861afe918763cbf33

Using the remap_stats plugin in production we have observed the remap_stats::stats_add() TSStatsCreate() getting the id from another stats. This happens only after a hard ATS crash and restart by traffic_manager. We've never seen an invalid id assigned, just an id from another stat. This may result in a bytes transferred counter incrementing a status_5xx like so (from traffic_ctl metric match remap_stats)

...
plugin.remap_stats.foo1.status_2xx 82704
plugin.remap_stats.loopback.status_5xx 727229625

NOTE: We've found that making the remap_stats persistent (param '-p') seems to mitigate whatever race condition there is.

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.

traeak commented 1 year ago

With removal of traffic_manager this might be resolved.