search

virtio-win / kvm-guest-drivers-windows

Windows paravirtualized drivers for QEMU\KVM
https://www.linux-kvm.org/page/WindowsGuestDrivers
BSD 3-Clause "New" or "Revised" License
2.02k stars 386 forks source link

virtio balloon virtio-win-0.1.190-1 does not release host memory #556

Closed jshen28 closed 3 years ago

jshen28 commented 3 years ago

with the the following packages installed,

Virtio balloon driver is installed and blnsvr service is running

Host memory does not get shrinked after instance boots up, and on the host rss is equal to the 16G which is the maximum

root@cmp004:~# top -p 8160
top - 17:28:03 up 20 days, 23:20,  1 user,  load average: 7.33, 8.07, 7.40
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s): 15.1 us,  8.8 sy,  0.0 ni, 75.8 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem : 13188148+total, 13976644 free, 10354286+used, 14361972 buff/cache
KiB Swap:  8388604 total,  6985204 free,  1403400 used. 23959712 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                               
 8160 libvirt+  20   0 18.352g 0.016t  31988 S 100.3 12.9 251:29.41 qemu-system-x86

but in the vm, the memory usage is (内存 means memory) is much less than 16 GB

image

libvirt config:

<memballoon model='virtio'>
  <stats period='10'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</memballoon>
YanVugenfirer commented 3 years ago

Hello @shen28 ,

Can you please send full steps from the host side? How are you inflating the balloon?

Best regards, Yan.

jshen28 commented 3 years ago

Hello, @YanVugenfirer ,

thank you very much for reply. We created a virtual machine using OpenStack, which basically used libvirt api internally. Actually we didn't manually inflate the balloon, the screen shot is taken right after vm booted up.

Best, Norman

in case it is needed, the xml looks like this

<domain type='kvm' id='3571'>
  <name>instance-000114b6</name>
  <uuid>13e03a60-2bf8-4ad3-95dd-8fedfdc485d7</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="17.0.9"/>
      <nova:name>win2012_mem01</nova:name>
      <nova:creationTime>2021-02-24 07:13:25</nova:creationTime>
      <nova:flavor name="ecs_2C16G260G_general">
        <nova:memory>16384</nova:memory>
        <nova:disk>260</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>2</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="f88e161b7adc4faf85920ae28a76c0ed">admin</nova:user>
        <nova:project uuid="f5a47e9639124287adf77b17d374212a">admin</nova:project>
      </nova:owner>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <shares>2048</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>17.0.9</entry>
      <entry name='serial'>3ceb1ccb-e574-4133-84a4-8ceb6fccaaaa</entry>
      <entry name='uuid'>13e03a60-2bf8-4ad3-95dd-8fedfdc485d7</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-bionic'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='1' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='97682fcf-f08f-49f4-ba09-b03da7325545'/>
      </auth>
      <source protocol='rbd' name='volumes/volume-20d4a118-fc66-44eb-b58c-ccec40843d5c'>
        <host name='172.16.2.61' port='6789'/>
        <host name='172.16.2.62' port='6789'/>
        <host name='172.16.2.63' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <serial>20d4a118-fc66-44eb-b58c-ccec40843d5c</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <auth username='nova'>
        <secret type='ceph' uuid='1c839bca-2a69-43dd-a0ad-6bff29684548'/>
      </auth>
      <source protocol='rbd' name='vms/13e03a60-2bf8-4ad3-95dd-8fedfdc485d7_disk.config'>
        <host name='172.16.2.61' port='6789'/>
        <host name='172.16.2.62' port='6789'/>
        <host name='172.16.2.63' port='6789'/>
      </source>
      <target dev='vdb' bus='virtio'/>
      <iotune>
        <total_bytes_sec>104857600</total_bytes_sec>
        <total_iops_sec>1000</total_iops_sec>
      </iotune>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:69:b1:46'/>
      <source bridge='qbr092a7099-96'/>
      <bandwidth>
        <inbound average='1048576'/>
        <outbound average='1048576'/>
      </bandwidth>
      <target dev='tap092a7099-96'/>
      <model type='virtio'/>
      <mtu size='1500'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/79'/>
      <log file='/var/lib/nova/instances/13e03a60-2bf8-4ad3-95dd-8fedfdc485d7/console.log' append='off'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/79'>
      <source path='/dev/pts/79'/>
      <log file='/var/lib/nova/instances/13e03a60-2bf8-4ad3-95dd-8fedfdc485d7/console.log' append='off'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-000114b6.sock'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5978' autoport='yes' listen='172.16.1.91' keymap='en-us'>
      <listen type='address' address='172.16.1.91'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+117</label>
    <imagelabel>+64055:+117</imagelabel>
  </seclabel>
</domain>
YanVugenfirer commented 3 years ago

@jshen28

Hi Norman,

I don't know how OpenStack should take advantage of the ballooning. It looks that the documentation is outdated https://wiki.openstack.org/wiki/Documentation/HypervisorTuningGuide : "libvirt/KVM has memory ballooning support, though Nova does not take advantage of it." In any case, from the driver's point of view, all the infrastructure is in place. But the management layers (Nova in OpenStack) didn't inflate the balloon.

Best regards, Yan.

jshen28 commented 3 years ago

@YanVugenfirer

Hi Yan,

thank you very much for the feedback. Actually OpenStack now supports balloon (please look at the xml generated which contains memballoon) and it works for linux vms..

Best,

Norman

vrozenfe commented 3 years ago

@jshen28

Can you try inflating/deflating balloon manually? What "info balloon" says?

Cheers, Vadim.

jshen28 commented 3 years ago

@vrozenfe

Hi Vadim,

thank you very much for the feedback, hope the following result is useful

# ps -ef | grep instance-000114b6
libvirt+ 37028     1  1 Mar01 ?        00:17:13 qemu-system-x86_64 -enable-kvm -name guest=instance-000114b6,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3636-instance-000114b6/master-key.aes -machine pc-i440fx-bionic,accel=kvm,usb=off,dump-guest-core=off -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -m 16384 -realtime mlock=off -smp 2,sockets=1,cores=1,threads=2 -uuid 13e03a60-2bf8-4ad3-95dd-8fedfdc485d7 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=17.0.9,serial=5adff07b-7ccf-43b9-bb39-f86d59cf6ebb,uuid=13e03a60-2bf8-4ad3-95dd-8fedfdc485d7,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-3636instance-000114b6/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -object secret,id=virtio-disk0-secret0,data=zX5G1uS1Vp692vfFTGw5H43im1xGtiNFG05xP8bgM30=,keyid=masterKey0,iv=DXkFCX8RcEo5bvt9os24pA==,format=base64 -drive file=rbd:volumes/volume-20d4a118-fc66-44eb-b58c-ccec40843d5c:id=cinder:auth_supported=cephx\;none:mon_host=172.16.2.61\:6789\;172.16.2.62\:6789\;172.16.2.63\:6789,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,serial=20d4a118-fc66-44eb-b58c-ccec40843d5c,cache=writeback,discard=unmap -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -object secret,id=virtio-disk1-secret0,data=gsDBTJUCtfvV9XFFCaK82c0Yei0xx97sSgYIfU5y/iI=,keyid=masterKey0,iv=CrCVvCCT1chuY0v2pbbQ5g==,format=base64 -drive file=rbd:vms/13e03a60-2bf8-4ad3-95dd-8fedfdc485d7_disk.config:id=nova:auth_supported=cephx\;none:mon_host=172.16.2.61\:6789\;172.16.2.62\:6789\;172.16.2.63\:6789,file.password-secret=virtio-disk1-secret0,format=raw,if=none,id=drive-virtio-disk1,cache=writeback,throttling.bps-total=104857600,throttling.iops-total=1000 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=48,id=hostnet0,vhost=on,vhostfd=52 -device virtio-net-pci,host_mtu=1500,netdev=hostnet0,id=net0,mac=fa:16:3e:69:b1:46,bus=pci.0,addr=0x3 -add-fd set=2,fd=90 -chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-000114b6.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 172.16.1.91:56 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on

# virsh qemu-monitor-command --hmp instance-000114b6 'info balloon'
balloon: actual=16384

# ps -o comm,pid,rss,vsz -f 37028 
COMMAND           PID   RSS    VSZ
qemu-system-x86 37028 17020928 19172220

# virsh qemu-monitor-command --hmp instance-000114b6 'balloon 8192'

# virsh qemu-monitor-command --hmp instance-000114b6 'info balloon'
balloon: actual=8192

# ps -o comm,pid,rss,vsz -f 37028 
COMMAND           PID   RSS    VSZ
qemu-system-x86 37028 8632316 19172220

After executing balloon command, the in-guest memory usage is shown below (hopefully my translation makes some sense...)

image

Indeed balloon command could shrink rss usage on the host, but why it uses up all rss right after guest boots up?

Best,

Norman

vrozenfe commented 3 years ago

@jshen28

Hi,

So it seems to be working fine.

"rss" went down "committed" (not "submitted") went up. "available" is also seems to correct as well.

Cheers, Vadim.

jshen28 commented 3 years ago

@vrozenfe

Hello,

kind of. but why windows vm seems uses up all rss during boot up? for linux vm, seems rss grows gradually.

Best, Norman

vrozenfe commented 3 years ago

Do you mean that balloon is always deflated after reboot?

If so, then balloon is always deflating when balloon device goes to D3 state https://github.com/virtio-win/kvm-guest-drivers-windows/blob/master/Balloon/sys/Device.c#L481

Cheers, Vadim.

jshen28 commented 3 years ago

@vrozenfe hmm.. not quite. what I mean is that when windows vm boots up, it seems to eat up all rss it's assigned. But for linux vm, when vm boots up, its host rss will start at a small value and most of time does not really grows as long as guest linux os does not ask for. (there is no manual inflate/deflate involved)

I would like to ask what causes this different and is there a way to make guest windows behaves more like guest linux os such that host rss could start from a small value.

Best,

Norman

vrozenfe commented 3 years ago

I guess because Windows zeroing out free pages???

jshen28 commented 3 years ago

@vrozenfe hmm... another interesting observation is that after live migration windows vm's rss got shrinked but if balloon xxx executed, host rss will immediately grows to the maximum...

vrozenfe commented 3 years ago

when you run balloon xxx do you inflate or deflate it?

vrozenfe commented 3 years ago

In any case balloon driver itself is a kind of passive element. It doesn't make any decision, just tries to fulfill a request coming from qemu to allocate more non-paged memory from the system or return it back.

jshen28 commented 3 years ago

@vrozenfe after live migration, if balloon manually inflated, looks like host rss will grow to the max. For example, after migration vm (16GB ram defined in libvirt xml) takes 2GB rss, executing balloon 8192 will cause host rss to immediately becomes 8GB, and in guest used memory becomes 8GB.

vrozenfe commented 3 years ago

@jshen28

Was the balloon inflated before migration? To what size?

jshen28 commented 3 years ago

@vrozenfe no, balloon is not inflated/deflated before migration.

vrozenfe commented 3 years ago

@jshen28

It is definitely above my level of expertise, so I don't know the right answer, I an only guessing. In my understanding rss is roughly equal to qemu memory + guest "in use" memory + video memory. So I guess when VM is idle, rss should go down. Live migration deals with VM and video dirty memories. But after migration, when you try to do ballooning, it is just triggers the massive memory manager activity, which in turns leads to increasing rss.

jshen28 commented 3 years ago

@vrozenfe thank you very much for your help. I'll try to do some more investigations.. close this as it is expected behavior.

yangni2017 commented 2 weeks ago

@vrozenfe thank you very much for your help. I'll try to do some more investigations.. close this as it is expected behavior.

hi, I have the same problem, do you have a solution now?