OOM crashes in v0.20.1

[plwhite]

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened:

Using 0.20.1 on Azure, all VMIs failed. The volumecontainerdisk container was hitting the OOM killer 100% of the time.

What you expected to happen:

VMIs to start!

How to reproduce it (as minimally and precisely as possible):

Take a vanilla ubuntu 18.04 image, and create a VMI with a minimal config. It will fail if you are running in Azure (AKS). Dockerfiles and VMI spec included below for reference.

Anything else we need to know?:

• Problem is specific to Azure; not seen elsewhere. Not a clue why, frankly, though if we are just running out of memory there are various possibilities.

• Problem is specific to v0.20.1; does not repro in v0.19.

• Problem manifests with either scratch containers or those based on kubevirt/container-disk-v1alpha

• If you create a VMI, then copy and hack the pod spec to remove the resource limits, then the OOM killer does not kick in; however the compute pod doesn't work either. This is not a supported thing to do but is suggestive that the underlying problem can be resolved with different limits for the volumecontainerimage pod.

Environment:

• KubeVirt version (use virtctl version): v0.20.1
• Kubernetes version (use kubectl version): 1.13.9
• VM or VMI specifications: See below
• Cloud provider or hardware configuration: Azure AKS, using nodes with 16/32 cores and 64/132GB RAM
• OS (e.g. from /etc/os-release): Ubuntu 16.04 as installed by AKS
• Kernel (e.g. uname -a): Don't know - getting access to the boxes is a pain, but could find out if it helps.
• Install tools: AKS

Dockerfile (using completely vanilla downloaded ubuntu image)

FROM kubevirt/container-disk-v1alpha
# qcow2
ADD bionic-server-cloudimg-amd64.img /disk

Also tried this Dockerfile:

FROM scratch
# qcow2
ADD bionic-server-cloudimg-amd64.img /disk

VMI spec I used

apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachineInstance
metadata:
  name: ubuntuvmi
  labels:
    billy: bob
spec:
  domain:
    resources:
      requests:
        memory: 1024M
        cpu: 2
    devices:
      disks:
      - name: containerdisk
        disk: {}
      - name: cloudinitdisk
        disk: {}
  volumes:
  - name: containerdisk
    containerDisk:
      image: metaswitchglobal.azurecr.io/plw/ubuntu:1804
  - name: cloudinitdisk
    cloudInitNoCloud:
      userData: |-
        #cloud-config
        password: ubuntu
        chpasswd: { expire: False }
        ssh_pwauth: True

I've discussed on slack with @slintes, @fabiand and @rmohr. Many thanks to all of you for your help.

[fabiand]

Peter, thanks for reporting. Was there an outcome of the discussion which gives a hint why this could be happening?

[plwhite]

Sorry for the delay, missed this question.

It looks like (slightly speculatively) the 20M resource limit in the volumecontainerdisk container is too low, and the OOM killer destroys the container. I don't have any idea why I always see the OOM killer and it isn't seen in other environments, but I am running in Azure.

If it's easy to get a build with a larger (or configurable) limit, it would be pretty simple for me to test a fix in that environment.

[fabiand]

Hey. it would be helpful if you can try to set a higher request for the container and see if it fixes your issue. There were some changes in the containerDisk area which could explain this.

[plwhite]

I thought the request was hard-coded in KubeVirt? I tried just manually changing the pod spec (removing limits from the containers), and that changed the behaviour, in that the VM didn't start any more but it didn't hit the OOM killer while copying disk contents.

[slintes]

yes, memory request/limit for the containerdisk container is hardcoded We might have ~2~ 1 issue~s~ here:

• the values might be too low for new containerdisks (strange though that this is happening in @plwhite's env only so far) ~- the values are very likely too low for old containerdisks~

[slintes]

the values are very likely too low for old containerdisks

oh, no, old containerdisks also use the new binary only

[plwhite]

To progress this, is there some way I can change the hardcoded limit? Even if it's a bit of a hack for now, it might give some information about what is going on.

[slintes]

You could build KubeVirt on your own, you only need to have make and docker installed:

• change limits at [0] and following lines
• run DOCKER_PREFIX="index.docker.io/<yourDockerUsername>" DOCKER_TAG="<aDockerTag>" make bazel-push-images manifests

That will build and push new images (make sure you are logged in in your docker account), and create manifests in _out/manifests/release

[0] https://github.com/kubevirt/kubevirt/blob/master/pkg/container-disk/container-disk.go#L126

[plwhite]

Thanks - let me have a shot at that. I should get to that later in the week.

[plwhite]

OK, so I changed that file and set all of the memory limits in the container to 200M and the CPU limits to 100m just for good measure. Having done so, I rebuilt and redeployed and can now create VMs based on container images - I've tried at least a dozen including some multi-GB images.

I have no idea what the actual memory limit should be in practice; for the monster images I am often working with 200M is a rounding error. Incidentally, this may be a silly proposal (I haven't got the backgorund), but I would have expected the volumecontainerdisk image to run as an init container and then terminate after the copy releasing that memory, in which case having a slightly over-large limit would matter much less.

[slintes]

Thanks a lot for the valueable feedback! I'm a bit busy myself this week, followed by 2 weeks vacation. Maybe @rmohr can follow up on this next week, when he is back from his vacation, he is very familiar with this topic.

[plwhite]

Sure - happy to help, especially given that this is after all a bug that only I have come across! Do let me know if there's anything else I can do.

[booxter]

@plwhite could you please retest with https://github.com/kubevirt/kubevirt/pull/2687 and close if no longer reproducible?

[plwhite]

Looks like #2687 fixes this.

(To be clear - I have tested and it no longer repros with the master build today, so definitely looks like a good fix.)

[slintes]

Great :) Thx for the feedback!

[terinjokes]

We saw this in 0.21.0. Was resolved after upgrading to 0.22.0.