pkramme
commented
1 month ago We threw together a quick script to visualize that problem:
0: seg18-app1, seg18-mysql1
1: seg17-app1, seg18-lb1
2: seg19-app1, seg19-redis1
3: seg19-mysql1
4: seg17-app1
5: seg17-mysql1, seg18-redis1
6: seg18-redis1, seg19-mysql1
7: seg19-app1, seg19-mysql1
8: seg18-app1, seg19-app1
9: seg18-app1, seg19-app1
10: seg17-lb1, seg18-mysql1
11: seg19-mysql1
12: seg19-app1, seg19-redis1
13: seg18-app1, seg18-mysql1
14: seg17-app1, seg17-redis1
15: seg18-mysql1
16: seg19-mysql1
17: seg19-app1, seg19-lb1
18: seg17-app1, seg17-redis1
19: seg17-mysql1, seg19-app1
20: seg18-app1, seg18-mysql1
21: seg19-mysql1
22: seg18-app1, seg18-mysql1
23: seg19-mysql1
24: seg19-mysql1
25: seg17-lb1, seg18-mysql1
26: seg18-mysql1
27: seg17-app1, seg18-app1
28: seg17-app1, seg18-app1
29: seg17-app1, seg19-app1
30: seg17-app1, seg18-lb1
31: seg19-lb1Cores:
- 0-7, 16-23 are fast
- 8-15, 24-31 are slow
- 16-31 are the same physical core to $index-16
General rule with this system is:
- seg17 has pratically no usage
- seg18 and seg19 are really important
This core placement puts very latency critical systems on the same (hyper)core, while leaving systems that have no real load on their own core. Even if this was a completely symmetrical CPU and even if all of those cores weren't hypercores, this would still waste resources when those VMs aren't equally loaded.
tomponline
Required information
The output of "lxc info" or if that fails:
Issue description
The introduction of automatic core scheduling has led to significant decrease in performance in our infrastructure, with weird problems that make no sense if you are not aware of this issue, such as:
The current CPU scheduler doesn't seem to understand hardware topology, which is really surprising, considering that many new CPUs are now asymmetric and that on the kernel side much work is being done making sure that workload is put on "the best core for the job" with features like AMD Preferred Core and equivalents or new CPU schedulers like EEVDF.
It seems weird to put these placement decisions in LXD and turn them on by default, without offswitch when LXD does not consider that this might cause significant problems. LXD simply has not enough data, and static round robin placement is simply too simple.
From our perspective this is a significant design error for this feature, and we ask that this feature is either
Additionaly, snaps auto update mechanism has introduced this new feature to our infrastructure (which by itself is fine), and we'd ask you to consider that features like this are being continously applied to real workloads and while not being LTS, should still be at least not harmful.
Information to attach
Our current hardware topology has two L3 domains that have different sizes. Our VMs run typical web application workloads. The core load balancing has put multiple CPU bound cores on one physical core, leading to the weird stealtime above.