search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
2.11k stars 1.11k forks source link

Incorrect `memoryintfreekbs` in listVirtualMachines for KVM #4566

Closed davidjumani closed 3 years ago

davidjumani commented 3 years ago

The memoryintfreekbs retuned by listVirtualMachines for KVM is always greater than the total memory allocated for the VM. This is because it returns the RSS of the process running the VM (https://libvirt.org/manpages/virsh.html#dommemstat) The RSS shows how much memory is allocated to that process and is in RAM. It is returned by memoryStats over at https://github.com/apache/cloudstack/blob/master/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java#L3712-L3718 when trying to fetch the free memory instead

Need to discuss how to handle this as well as the impact on usage

ISSUE TYPE
CLOUDSTACK VERSION
4.11+ (tested)
OS / ENVIRONMENT

KVM

STEPS TO REPRODUCE
(localcloud) SBCM5> > list virtualmachines filter=memory,memoryintfreekbs,memorykbs,memorytargetkbs,
{
  "count": 3,
  "virtualmachine": [
    {
      "memory": 512,
      "memoryintfreekbs": 552836, <---
      "memorykbs": 524288,
      "memorytargetkbs": 524288
    },
EXPECTED RESULTS
The used memory
ACTUAL RESULTS
Total memory used by the process running the VM
davidjumani commented 3 years ago

Related to https://github.com/apache/cloudstack/issues/3456

DaanHoogland commented 3 years ago

@PaulAngus @andrijapanicsb or any other functional eyes on this (@weizhouapache, @wido maybe from your camp), This one raises a question about billing, should an operator be able to see both memory used by the vm and by the containing process for their accounting? what is the best un-ambiguous way to define this?

PaulAngus commented 3 years ago

This issue jumps between FREE memory and USED memory - which one are we talking about here?

DaanHoogland commented 3 years ago

the reported issue is about how free mem is calculated but the real issue is whether used memory as used by the VM-process as opposed to the memory used inside the vm should be attributed to the user or reported in some way at all

PaulAngus commented 3 years ago

thanks. can we change the title to match the problem please. Anyone looking back through commits will be misled.

I don't think that the overhead to run a VM is the end-users problem, I think that if I created a VM with 8GB RAM, I would expect to get charged for 8GB.

wido commented 3 years ago

@PaulAngus I agree. Overhead is something the operator should take into account. We at PCextreme do that. We take a safety margin on top of all the memory we allocate to VMs and which we then reserve for the Operating System, buffers and overhead.

So yes, the end-user should never be aware of the RSS on the hypervisor.

VMs using Ceph for example have a much higher overhead to due librados and librbd consuming memory under the hood.

andrijapanicsb commented 3 years ago

agree with Paul and Wido.

davidjumani commented 3 years ago

I agree as well so I gather that we have a consensus on it. The other issue is that the memoryintfreekbs parameter returns the wrong result (the rss instead) and it might not be possible to retrieve the free memory in the VM. What would be a good approach to go about handling it ? @PaulAngus @wido @andrijapanicsb @DaanHoogland

wido commented 3 years ago

Inside the VM depends on the hypervisor I guess. With KVM for example you have the Qemu Guest Agent which is needed to look inside the VM and get these results.

I don't think we can safely fetch that information from inside the VM.

andrijapanicsb commented 3 years ago

+1 on what Wido said, as this is public cloud people will be (mostly/soemtimes) running, so it's impossible to ensure specific hypervisor tools are running. I don't see the value of monitoring the amount of RAM used inside the VM - let me user monitor his own VMs if needed (not the operator) using whatever tools they use.

weizhouapache commented 3 years ago

thanks. can we change the title to match the problem please. Anyone looking back through commits will be misled.

I don't think that the overhead to run a VM is the end-users problem, I think that if I created a VM with 8GB RAM, I would expect to get charged for 8GB.

Totally agree.

by the way, there is a way to get accurate memory usage: memory ballooning. see https://libvirt.org/formatdomain.html#memory-balloon-device

  <devices>
    <memballoon model='virtio'>
      <stats period='10'/>
    </memballoon>
  </devices>

we could add a global setting to enable/disable it.

DaanHoogland commented 3 years ago

ok, i will not make a point of this but as an enterprise cloud operator I am not at all interested in the memory usage of a vm/user, only in the amount of memory it costs me/the company. I understand public cloud wins here.

davidjumani commented 3 years ago

So since the field's docs mention that it is the free memory inside the VM, guessing the docs should be changed or set to zero for KVM ? Right now it is greater than the total configured memory and used in the new UI while displaying the VM details

wido commented 3 years ago

thanks. can we change the title to match the problem please. Anyone looking back through commits will be misled. I don't think that the overhead to run a VM is the end-users problem, I think that if I created a VM with 8GB RAM, I would expect to get charged for 8GB.

Totally agree.

by the way, there is a way to get accurate memory usage: memory ballooning. see https://libvirt.org/formatdomain.html#memory-balloon-device

  <devices>
    <memballoon model='virtio'>
      <stats period='10'/>
    </memballoon>
  </devices>

we could add a global setting to enable/disable it.

Personally I'm not really a fan of all the global settings we are introducing in CloudStack. The list gets longer and longer.

That said, virtio ballooning is also something which needs to be supported by the guest. Linux does this, but Windows or BSD won't always do this.

wido commented 3 years ago

So since the field's docs mention that it is the free memory inside the VM, guessing the docs should be changed or set to zero for KVM ? Right now it is greater than the total configured memory and used in the new UI while displaying the VM details

I would set it to 0 for KVM indeed. We can't reliably get this data.

davidjumani commented 3 years ago

Added a fix to set memoryintfreekbs to zero if it is greater than memorykbs

weizhouapache commented 3 years ago

thanks. can we change the title to match the problem please. Anyone looking back through commits will be misled. I don't think that the overhead to run a VM is the end-users problem, I think that if I created a VM with 8GB RAM, I would expect to get charged for 8GB.

Totally agree. by the way, there is a way to get accurate memory usage: memory ballooning. see https://libvirt.org/formatdomain.html#memory-balloon-device

  <devices>
    <memballoon model='virtio'>
      <stats period='10'/>
    </memballoon>
  </devices>

we could add a global setting to enable/disable it.

Personally I'm not really a fan of all the global settings we are introducing in CloudStack. The list gets longer and longer.

yep. but cloudstack is a open source project. There are some features which are used by some users but not used by other users. They might use different setting. It is the certain result if more features and improvements are added.

That said, virtio ballooning is also something which needs to be supported by the guest. Linux does this, but Windows or BSD won't always do this.

@wido true. In windows, user have to install virtio driver for memory ballooning. If the driver is installed incorrectly, vm might be frozen after migration.

rohityadavcloud commented 3 years ago

Fixed in https://github.com/apache/cloudstack/pull/4571