Open tstarling opened 7 years ago
cc @jasone
This happened to us a year ago, and we brought it up with @jasone via email. His explanation at the time:
At Facebook we configured jemalloc with only one automatic arena starting a couple of years ago because for our workloads at the time, we measured no significant decrease in throughput, but a significant decrease in memory usage thanks to fragmentation reduction. In general I think this is a dangerous default because of the potential for issues like the one you (Etsy) hit, and as it happens we are currently running experiments to determine whether
narenas:1
is still appropriate for Facebook workloads.
To which @jwatzman replied:
Unless Jason thinks otherwise... let's wait a couple weeks, for the lockdown experiments on this to conclude; if we end up using a value more reasonable outside of FB than just "1", we may not want to add a knob for this, since it's so so specialized and we already have too many knobs.
Perhaps those experiments illuminated a reason for keeping it set to 1?
My fuzzy recollection is that increasing the number of automatic arenas had little impact on Facebook workload throughput. For Facebook, narenas:1
works well, but IMO it's not ideal as a generic default.
Since 7798145279de60f285173d9a2fd4ca9025b32db8 (Oct 2012), HHVM overrides jemalloc's number of arenas (narenas), setting it to 1. The default is 4 times the number of CPUs. In WMF bug T151702 we are considering an overload incident which involved requests doing json_decode() calls on large JSON blobs with high concurrency. The json-c library called malloc() very frequently. Contention on one of the bin locks within the single jemalloc arena limited CPU usage to about 50%. Benchmarking suggests that the default would have worked just fine.
For a less demanding workload, I'm sure that setting it to 1 provides a small benefit in memory usage, and perhaps a CPU usage benefit would be detectable with careful testing. But I think this kind of workload-dependent tuning is best done in site configuration. The default should have reasonable performance over a wide variety of workloads, rather than pathologically limiting concurrency.
We are using HHVM 3.12.7. The benchmark data was collected on a server with 40 CPUs and 64GB of memory.