windows vm allocating all memory configured in currentMemory at startup

WosunOO commented 1 year ago

Describe the bug My windows vm is allocating all memory configured in currentMemory at startup. Changing currentMemory with virsh setmem or virsh qemu-monitor-command instance-00000001 --hmp "balloon 4096" can successfully change memory status in my windows vms, so the balloon service and balloon driver is working I think. Also checked the balloon driver's tracing events with TraceView.exe. Any help will be appreciated. Thank you!

To Reproduce Well, it's always appearing in my env.

Expected behavior As for linux vms in my env, they use memory much lesser than currentMemory configured in vm's xml (RES value seen with host's top command). When setting memory balloon linux vms RES value seen on host will decrese to which the vm really needs. But windows vms are not working this way. I did a lot of searching and found proxmox wiki on windows vm memory ballooning(https://pve.proxmox.com/wiki/Dynamic_Memory_Management#Requirements_for_Windows_VM), did proxmox changed the mechanism of qemu simulator?

Screenshots windows vm memory usage seen on host:

Host:

Disto: Ubuntu
Kernel version: 4.15.0-197 generic
QEMU version: 6.2.0
QEMU command line : /usr/bin/qemu-system-x86_64 -name guest=instance-00000001,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-98-instance-00000001/master-key.aes"} -machine pc-i440fx-6.2,accel=kvm,usb=off,dump-guest-core=off -cpu Broadwell-noTSX-IBRS,3dnowprefetch=off,adx=off,invtsc=off,mpx=off,rdseed=off,smap=off,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff -m size=10485760k,slots=255,maxmem=131900416k -overcommit mem-lock=off -smp 2,maxcpus=32,sockets=32,dies=1,cores=1,threads=1 -object {"qom-type":"memory-backend-ram","id":"ram-node0","size":10737418240} -numa node,nodeid=0,cpus=0-31,memdev=ram-node0 -uuid e7ff3192-bcb1-45a6-a79d-251aaa252645 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=22.0.2,serial=e7ff3192-bcb1-45a6-a79d-251aaa252645,uuid=e7ff3192-bcb1-45a6-a79d-251aaa252645,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=31,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -object {"qom-type":"secret","id":"libvirt-2-storage-auth-secret0","data":"****","keyid":"masterKey0","iv":"****","format":"base64"} -blockdev {"driver":"rbd","pool":"nova.vms","image":"e7ff3192-bcb1-45a6-a79d-251aaa252645_disk","server":[{"host":"192.168.8.5","port":"6789"},{"host":"192.168.8.3","port":"6789"},{"host":"192.168.8.2","port":"6789"}],"user":"cinder_a4842e0c-3207-4945-91af-99c47b3aa79e","auth-client-required":["cephx","none"],"key-secret":"libvirt-2-storage-auth-secret0","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","cache":{"direct":false,"no-flush":false}
libvirt version: 7.8.0

libvirt XML file:

<domain type='kvm' id='1'>
<name>instance-00000001</name>
<uuid>e7ff3192-bcb1-45a6-a79d-251aaa252645</uuid>
<metadata>
<nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
  <nova:package version="22.0.2"/>
  <nova:name>test_windows_mem_balloon</nova:name>
  <nova:creationTime>2023-08-14 09:42:04</nova:creationTime>
  <nova:flavor name="2C10G100G">
    <nova:memory>10240</nova:memory>
    <nova:disk>100</nova:disk>
    <nova:swap>0</nova:swap>
    <nova:ephemeral>0</nova:ephemeral>
    <nova:vcpus>2</nova:vcpus>
  </nova:flavor>
  <nova:owner>
    <nova:user uuid="31c4c5d2a06347029f7abadc79deee09">admin</nova:user>
    <nova:project uuid="7c350a8ca24b4a16a8687cc405756486">admin</nova:project>
  </nova:owner>
  <nova:root type="image" uuid="5db2e143-b6f1-4ce2-83f8-8fb0ecb639aa"/>
</nova:instance>
</metadata>
<maxMemory slots='255' unit='KiB'>131900416</maxMemory>
<memory unit='KiB'>10485760</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<vcpu placement='static' current='2'>32</vcpu>
<cputune>
<shares>2048</shares>
</cputune>
<resource>
<partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
<system>
  <entry name='manufacturer'>OpenStack Foundation</entry>
  <entry name='product'>OpenStack Nova</entry>
  <entry name='version'>22.0.2</entry>
  <entry name='serial'>e7ff3192-bcb1-45a6-a79d-251aaa252645</entry>
  <entry name='uuid'>e7ff3192-bcb1-45a6-a79d-251aaa252645</entry>
  <entry name='family'>Virtual Machine</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-6.2'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
  <relaxed state='on'/>
  <vapic state='on'/>
  <spinlocks state='on' retries='8191'/>
</hyperv>
</features>
<cpu mode='custom' match='exact' check='full'>
<model fallback='forbid'>Broadwell-noTSX-IBRS</model>
<topology sockets='32' dies='1' cores='1' threads='1'/>
<feature policy='disable' name='3dnowprefetch'/>
<feature policy='disable' name='adx'/>
<feature policy='disable' name='invtsc'/>
<feature policy='disable' name='mpx'/>
<feature policy='disable' name='rdseed'/>
<feature policy='disable' name='smap'/>
<feature policy='require' name='vme'/>
<feature policy='require' name='f16c'/>
<feature policy='require' name='rdrand'/>
<feature policy='require' name='hypervisor'/>
<feature policy='require' name='arat'/>
<feature policy='require' name='xsaveopt'/>
<feature policy='require' name='abm'/>
<numa>
  <cell id='0' cpus='0-31' memory='10485760' unit='KiB'/>
</numa>
</cpu>
<clock offset='localtime'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='network' device='disk'>
  <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
  <auth username='cinder_a4842e0c-3207-4945-91af-99c47b3aa79e'>
    <secret type='ceph' uuid='a4842e0c-3207-4945-91af-99c47b3aa79e'/>
  </auth>
  <source protocol='rbd' name='nova.vms/e7ff3192-bcb1-45a6-a79d-251aaa252645_disk' index='2'>
    <host name='192.168.8.5' port='6789'/>
    <host name='192.168.8.3' port='6789'/>
    <host name='192.168.8.2' port='6789'/>
  </source>
  <target dev='vda' bus='virtio'/>
  <alias name='virtio-disk0'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<disk type='network' device='disk'>
  <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
  <auth username='cinder_a4842e0c-3207-4945-91af-99c47b3aa79e'>
    <secret type='ceph' uuid='a4842e0c-3207-4945-91af-99c47b3aa79e'/>
  </auth>
  <source protocol='rbd' name='nova.vms/e7ff3192-bcb1-45a6-a79d-251aaa252645_disk.config' index='1'>
    <host name='192.168.8.5' port='6789'/>
    <host name='192.168.8.3' port='6789'/>
    <host name='192.168.8.2' port='6789'/>
  </source>
  <target dev='vdb' bus='virtio'/>
  <alias name='virtio-disk1'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>
<controller type='usb' index='0' model='piix3-uhci'>
  <alias name='usb'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
  <alias name='pci.0'/>
</controller>
<controller type='virtio-serial' index='0'>
  <alias name='virtio-serial0'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<interface type='bridge'>
  <mac address='fa:16:3e:3d:3e:00'/>
  <source bridge='qbrd3c1bec5-45'/>
  <target dev='tapd3c1bec5-45'/>
  <model type='virtio'/>
  <mtu size='1500'/>
  <alias name='net0'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
  <source path='/dev/pts/12'/>
  <log file='/var/lib/nova/instances/e7ff3192-bcb1-45a6-a79d-251aaa252645/console.log' append='off'/>
  <target type='isa-serial' port='0'>
    <model name='isa-serial'/>
  </target>
  <alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/12'>
  <source path='/dev/pts/12'/>
  <log file='/var/lib/nova/instances/e7ff3192-bcb1-45a6-a79d-251aaa252645/console.log' append='off'/>
  <target type='serial' port='0'/>
  <alias name='serial0'/>
</console>
<channel type='unix'>
  <source mode='bind' path='/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-00000001.sock'/>
  <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
  <alias name='channel0'/>
  <address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='tablet' bus='usb'>
  <alias name='input0'/>
  <address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'>
  <alias name='input1'/>
</input>
<input type='keyboard' bus='ps2'>
  <alias name='input2'/>
</input>
<graphics type='vnc' port='5900' autoport='yes' listen='192.168.8.1'>
  <listen type='address' address='192.168.8.1'/>
</graphics>
<audio id='1' type='none'/>
<video>
  <model type='cirrus' vram='16384' heads='1' primary='yes'/>
  <alias name='video0'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
  <stats period='10'/>
  <alias name='balloon0'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</memballoon>
<rng model='virtio'>
  <backend model='random'>/dev/urandom</backend>
  <alias name='rng0'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</rng>
</devices>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+42424:+109</label>
<imagelabel>+42424:+109</imagelabel>
</seclabel>
</domain>

VM:

Tried windows2012r2 and windows2016
Tried 11/15/2022, 100.92.104.22900(I also builded a driver myself with branch master which has the same issue)

WosunOO commented 1 year ago

I have read issue #568 carefully, behaviors seem to be the same. Also found that automatically changing memory balloon (currentMemory) is never completed on linux-kvm page. Main concerns are the difference between windows and linux vms on memory usage shown on host.

YanVugenfirer commented 1 year ago

Hello,

Is balloon service running? It is mentioned in Proxmox link.

WosunOO commented 1 year ago

Hello,

Is balloon service running? It is mentioned in Proxmox link.

Yes, blnsvr is installed and process running; Also tried different versions of blnsvr, all the same. Or maybe I should try older versions?

By the way, does Memory Management Registry affect this behavior, also tried turning off PAE etc...

YanVugenfirer commented 1 year ago

What I am missing is the host statistics when Windows VM is running with and without inflated balloon. Otherwise the difference between Linux \Window can be a natural difference between those OSes.

BTW: KSM is a good way to save memory with Windows VMs that are not too active because Windows is actively zeroing unused memory.

WosunOO commented 1 year ago

What I am missing is the host statistics when Windows VM is running with and without inflated balloon. Otherwise the difference between Linux \Window can be a natural difference between those OSes.

BTW: KSM is a good way to save memory with Windows VMs that are not too active because Windows is actively zeroing unused memory.

Thank you for replying. I just tried uninstalled blnsvr service and disabled virtio-balloon driver. Then my vm's memory cannot be set by balloon. On the host the memory occupation is always the maximum value, in my case 10GB. It works normal when enabled virtio-balloon driver, without installing blnsvrservice. Wondering what role is blnsvr service playing? So it's due to the difference between OSes? Have no idea how proxmox makes windows works just like linux vms. Have been exploring their git repo like pve-qemu, but didn't find any patch may related to this issue.

jshen28 commented 1 year ago

I am curious if windows zeros out all pages during booting up. Because if it is, then it is reasonable that Linux kernel will eventually allocates all pages.

izeroo commented 9 months ago

CurrentMemory should be set the same as MaxMemory unless you known what really you are doing. Inconsistent values can cause problem like this. I encounterd this problem as well.

virtio-win / kvm-guest-drivers-windows

windows vm allocating all memory configured in currentMemory at startup #964