search

xcp-ng / xcp

Entry point for issues and wiki. Also contains some scripts and sources.
https://xcp-ng.org
1.32k stars 74 forks source link

Real-Time: VCPU mask may not be able to ensure the real-time performance of the VM, but makes it worse. #586

Open DyingOwl opened 1 year ago

DyingOwl commented 1 year ago

Hi~ I am trying to migrate from XEN (original official version) to XCP-ng. When I tried to configure the virtual machine, I found it difficult to ensure its real-time performance. In my previous XEN environment, the most effective method is to use cpus = ['4','5'] parameter (for example) in xl.cfg file, which can bind vcpu0 and pcpu4, vcpu1 and pcpu5 of this VM. It can be confirmed in the following ways:

# xl vcpu-list test
name ID  VCPU  CPU  ...  Affinity (Hard / Soft)
test  1     0    4  ...  4 / all
test  1     1    5  ...  5 / all

With some other configurations in VM's grub and kernel, it can ensure that the maximum value measured by cyclictest -p 80 -t5 -n -i 1000 -l ... does not exceed 300us. In contrast, I found command xe vm-param-set uuid=[VM-UUID] VCPUs-params:mask='4;5' in XCP-ng, which can cause the same result as the previous. Before executing vm-param-set, the maximum test result of cyclictest is about 4ms, but it exceeds 15ms (sometimes even 25ms) after execution and reboot. According to historical experience, providing VNC functions for VM may also affect test results. So when I log in through SSH(instead of XO or XCP-ng Center), the maximum can drop to 10ms. After I temporarily disable the vga of the VM by modifying file /usr/lib64/xen/bin/qemu-wrapper, the maximum can drop to 2ms, but there is still a big gap with my previous environment.

In the XCP-ng environment, is VCPU mask designed to optimize the stability of VCPU scheduling and the real-time performance of the virtual machine? If not, how can I ensure that the virtual machine has high real-time performance in the XCP-ng environment?

olivierlambert commented 1 year ago

Hi,

Never heard of CPU masks before, only CPU pinning. I'm not sure about the difference with masks :thinking:

Have you tried pinning? See https://support.citrix.com/article/CTX246466/how-to-pin-citrix-hypervisor-virtual-cpus-to-specific-physical-cpus

DyingOwl commented 1 year ago

Hi,

Never heard of CPU masks before, only CPU pinning. I'm not sure about the difference with masks 🤔

Have you tried pinning? See https://support.citrix.com/article/CTX246466/how-to-pin-citrix-hypervisor-virtual-cpus-to-specific-physical-cpus

In my understanding, these are the same things (sometimes called NUMA affinity, either). I use this name in the title because the function needs to be configured through the VCPU mask parameter in the xe command. Refer doc