support kvm run-time memory changes (ballooning)

[GoogleCodeExporter]

What software version are you running? Please provide the output of "gnt-
cluster --version", "gnt-cluster version", and "hspace --version".

gnt-cluster version 
Software version: 2.9.2
Internode protocol: 2090000
Configuration format: 2090000
OS api version: 20
Export interface: 0
VCS version: v2.9.2

hspace (ganeti) version v2.9.2
compiled with ghc 7.4
running on linux x86_64

What distribution are you using?

ubuntu 12.04.4 lts

What steps will reproduce the problem?
1. gnt-instance modify -B memory=1G instancename
2. run free -m on the instance

What is the expected output? What do you see instead?

ganeti should run an qemu monitor command to notify kvm about the memory change:

echo "balloon 1024" | nc -U 
/var/run/ganeti/kvm-hypervisor/ctrl/instance1.fh-koblenz.de.monitor

current qemu/kvm ballon setting can be shown with this command:

echo "info balloon" | nc -U 
/var/run/ganeti/kvm-hypervisor/ctrl/instance1.fh-koblenz.de.monitor

Original issue reported on code.google.com by hei...@googlemail.com on 29 Jan 2014 at 8:45

[GoogleCodeExporter]

Original comment by aeh...@google.com on 29 Jan 2014 at 9:27

• Changed state: Accepted
• Added labels: Milestone-Release2.12, Type-Enhancement

[GoogleCodeExporter]

hmm i found "gnt-instance --runtime-memory" which changes the memory settings 
of a running kvm instance and it works fine. inside the instance a "free -m" 
shows the right amount of ram. but this setting is lost after the instance is 
shutdown.

so i thought that it is the right was to configure memory settings like this:

root@vserver8:~# gnt-instance modify -B memory=1G,maxmem=10G instance1 
Modified instance instance1
 - be/memory -> 1024
 - be/maxmem -> 10240
Please don't forget that most parameters take effect only at the next (re)start 
of the instance initiated by ganeti; restarting from within the instance will 
not be enough.

but this does not work:

root@vserver8:~# gnt-instance info instance1 | grep mem
    mem_path: default ()
    maxmem: 1024
    memory: default (1024)
    minmem: 1024

maxmem always is set to the size of memory.

i guess it should be something like this?

minmem = lower limit which could be configured while instance is runngin (with 
--runtime-memory) 
maxmem = upper limit which could be configured while instance is running (with 
--runtime-memory)
memory = the amount of memory the instance get on startup

for kvm it seems like that ganeti should start the instance with "kvm -m maxmem 
...." and decrease memory after instance startup to "memory" like 
"--runtime-memory" does?

it looks like kvm does not know a maxmem option and -m sets the upper limit?

would be great to have such a behaviour.

or do i misunderstand the memory concepts of ganeti?

Original comment by hei...@googlemail.com on 30 Jan 2014 at 12:14

[GoogleCodeExporter]

Hi,

This is working as intended. You can either set "memory" (which is a single 
value) or maxmem and minmem. memory and maxmem together won't make sense. If 
you use maxmem and minmem ganeti will always start the instance with as much 
memory as it can. --runtime-memory can be used to adjust this value.

Thanks,

Guido

Original comment by ultrot...@google.com on 30 Jan 2014 at 1:14

[GoogleCodeExporter]

hmm. but what if i want an instance to have for example 2G per default (on 
startup) but the ability to increase it to a max of 10G on demand?

if i understand the current behaviour right this means i have to run two 
commands on startup?

gnt-instance start instance1
gnt-instance --runtime-memory 2G instance1

but this way the "runtime-" or "startup-memory" setting is not stored in the 
cluster config. does it not make more sense to have it like i described above? 
this is much like the xen maxmem config file parameter works:

maxmem = upper limit
memory = startup memory

for my understanding it would make sense to have a separate startup memory 
setting for each instance in the cluster config.

or what do you think?

thanks
Heiko

Original comment by hei...@googlemail.com on 30 Jan 2014 at 1:44

[GoogleCodeExporter]

Here is how libvirt handels memory settings:

http://libvirt.org/formatdomain.html#elementsMemoryAllocation

Its the same as with xen. You can set the maximum memory and the 
current/startup memory for an instance.

Original comment by hei...@googlemail.com on 3 Feb 2014 at 7:24

[GoogleCodeExporter]

and here the xen documenation for memory settings:

http://wiki.xen.org/wiki/Xen_Configuration_File_Options#VARIABLES

Original comment by hei...@googlemail.com on 3 Feb 2014 at 7:56

[GoogleCodeExporter]

As you see this is similar to Xen's "maxmem" and "memory", where we use our 
maxmem as maxmem, and the maximum available memory as memory. What's missing is 
a dynamic way to reduce it all the way down to minmem when it's needed. Do you 
have exact requirements of what you'd like to happen? And how would you change 
things to make that possible?
(note that it's better if we start from requirements than from the actual 
implementation)

Thanks,
Guido

Original comment by ultrot...@google.com on 4 Feb 2014 at 12:41

[GoogleCodeExporter]

what does "a dynamic way to reduce" mean?

manual runtime changes can already be made with --runtime-memory if i am right.

an simple use case is to have an instance started with e.g. 2G and set it to 8G 
while compiling a something.

i think the best would be like i described in #1

minmem = lower limit which could be configured while instance is runngin (with 
--runtime-memory) 
maxmem = upper limit which could be configured while instance is running (with 
--runtime-memory)
memory = the amount of memory the instance get on startup

or is this not possible for some reason?

thanks
heiko

Original comment by hei...@googlemail.com on 4 Feb 2014 at 1:19

[GoogleCodeExporter]

This would be nice to have.

In our implementation we need something like this and I was planning on putting 
a tag on the instance and set the memory via a hook or something like that.

But of course, a native solution would be ideal.

Original comment by ricardo....@gmail.com on 4 Feb 2014 at 4:08

[GoogleCodeExporter]

Indeed, that would be a nice (and expected) behavior. 

Dynamic growing/shrinking depending on various scenarios (memory pressure, 
instance about to be migrated, etc) are nice, but not a pre-requisite for this 
simple enhancement. 

The ability to manually increase memory at some point during run time w/o the 
need for a restart is what makes this request attractive.

Original comment by chibi0...@gmail.com on 5 Feb 2014 at 5:41

[GoogleCodeExporter]

I think it is confusing that gnt-instance returns any backend "memory" setting, 
and this was supposed to have been removed in ganeti 2.7:

[lib/client/gnt_instance.py]

  # deprecated "memory" value, kept for one version for compatibility
  # TODO(ganeti 2.7) remove.
  be_actual = copy.deepcopy(instance["be_actual"])
  be_actual["memory"] = be_actual[constants.BE_MAXMEM]

As you can see, the "memory" value returned is just a copy of the "maxmem" 
value, which is not useful. Maybe now is the time to remove it?

Related: issue #850. The "memory" setting has already been removed from 
config.ini when exporting an instance, leaving just "minmem" and "maxmem".

You *can* specify -B memory=xxx when modifying an instance, but as far as I can 
see, it just sets -B minmem=xxx,maxmem=xxx

Other comments: it sounds from this ticket that what's wanted is a setting for 
"the amount of memory to allocate on instance startup" - but how would that 
interact with the current algorithm that picks a value between minmem and 
maxmem? Does it essentially become an override for maxmem for all automatic 
ballooning operations?

Finally, I guess it would also be useful to have a way to see the *actual* 
memory allocated to the VM right now. This is similar to ticket #854, where the 
poster wants to be able to see the run-time values of parameters set in a 
one-shot way using gnt-instance start [-H/-B], which is different to the 
startup values stored against the instance.

Original comment by bcandler...@googlemail.com on 20 Jun 2014 at 9:50

[GoogleCodeExporter]

>Other comments: it sounds from this ticket that what's wanted is a setting for 
"the 
>amount of memory to allocate on instance startup" - but how would that 
interact 
>with the current algorithm that picks a value between minmem and maxmem? Does 
it 
>essentially become an override for maxmem for all automatic ballooning 
operations?

like described above it think would be the best to see "minmem" and "maxmem" 
only as the lower and upper limits for an running instance between which the 
runtime memory of an instance could be changed. and "memory" is used as startup 
memory allocation for an instance. thats how it works for xen and kvm it think.

http://libvirt.org/formatdomain.html#elementsMemoryAllocation
http://wiki.xen.org/wiki/Xen_Configuration_File_Options#VARIABLES

what i do not know if there is any automatic memory ballooning implemented in 
ganeti. but one case where automatic changing of runtime memory could be needed 
is when an instance needs to be migrated to an node where not enough memory is 
available. in this case minmem could be the lower limit to which ganeti should 
be allowed to automatically reduce the instance current/runtime memory.

maybe i miss something because i dont have the whole picture of ganetis 
behavior but for me this makes the most sense.

>Finally, I guess it would also be useful to have a way to see the *actual* 
memory 
>allocated to the VM right now. This is similar to ticket #854, where the 
poster 
>wants to be able to see the run-time values of parameters set in a one-shot 
way 
>using gnt-instance start [-H/-B], which is different to the startup values 
stored 
>against the instance.

yes. it think "gnt-instance list" should show the current memory of the 
instance.

Original comment by hei...@googlemail.com on 25 Aug 2014 at 7:43