Excessive memory usage in `minikube image load`

SpecLad commented 11 months ago

What Happened?

$ docker pull cvat/server:v2.9.2
[... snipped ...]
$ minikube --alsologtostderr image load cvat/server:v2.9.2
I1213 17:52:14.641171 1485897 out.go:296] Setting OutFile to fd 1 ...
I1213 17:52:14.647657 1485897 out.go:309] Setting ErrFile to fd 2...
I1213 17:52:14.647834 1485897 root.go:338] Updating PATH: /home/dpb/.minikube/bin
I1213 17:52:14.648351 1485897 config.go:182] Loaded profile config "minikube": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.28.3
I1213 17:52:14.648471 1485897 cache.go:107] acquiring lock: {Name:mke937cece9923eb59ace0008a0ae80e0c33e4c8 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I1213 17:52:14.648854 1485897 image.go:134] retrieving image: cvat/server:v2.9.2
I1213 17:52:14.648908 1485897 image.go:140] checking repository: index.docker.io/cvat/server
I1213 17:52:15.770377 1485897 image.go:144] canonical name: docker.io/cvat/server:v2.9.2
I1213 17:52:15.784553 1485897 image.go:173] found index.docker.io/cvat/server:v2.9.2 locally: &{ref:{Repository:{Registry:{insecure:false registry:index.docker.io} repository:cvat/server} tag:v2.9.2 original:cvat/server:v2.9.2} opener:0xc0003fd810 tarballImage:<nil> computed:false id:0xc0023cfcc0 configFile:<nil> once:{done:0 m:{state:0 sema:0}} err:<nil>}
I1213 17:52:15.784656 1485897 cache.go:162] opening:  /home/dpb/.minikube/cache/images/amd64/cvat/server_v2.9.2
Killed

Looking at dmesg output, I see this:

[246684.160979] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=minikube,pid=1463487,uid=1000
[246684.161006] Out of memory: Killed process 1463487 (minikube) total-vm:11663440kB, anon-rss:5821500kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:11772kB oom_score_adj:0

It seems like minikube is using an excessive amount of memory, more even than the size of the image itself (which is under 3 GB).

Attach the log file

log.txt

Operating System

Ubuntu

Driver

Docker

jusito commented 10 months ago

Experienced the same issues.

minikube version: v1.32.0 (identical to log above) commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d (identical to log above) Docker version 24.0.7, build afdd53b

image with 8.7GB in ~/.minikube/cache/image loaded with: minikube image load --daemon=true --remote=false --overwrite=false xxx With "--alsologtostderr" this also occurred during "opening".

afbjorklund commented 10 months ago

Do you get the same memory usage, if you load the image manually with the container runtime?

EDIT: never mind, this looks related to crane and the minikube cache. Probably have to use "docker save" first

jusito commented 10 months ago

I only understand your suggestion after your referenced it in the other post, can confirm the workaround seems to work fine.

Workaround: For others instead of minikube image load IMAGE use:

docker save --output docker.img IMAGE
minikube image load docker.img

Results:

improved elapsed time from 30 to 15 minutes
improved memory consumption from 40GB to 100MB

/usr/bin/time details: minikube image load IMAGE

Command being timed: "minikube image load compiling-environment:2024.01.12_16.08.20"
User time (seconds): 833.21
System time (seconds): 132.44
Percent of CPU this job got: 51%
Elapsed (wall clock) time (h:mm:ss or m:ss): 30:57.30
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 39197588
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 536
Minor (reclaiming a frame) page faults: 11303717
Voluntary context switches: 1579611
Involuntary context switches: 156191
Swaps: 0
File system inputs: 91256
File system outputs: 8738368
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

vs docker image save --output docker.img IMAGE

Command being timed: "docker image save --output current-docker.img compiling-environment:2024.01.12_16.08.20"
User time (seconds): 2.19
System time (seconds): 22.88
Percent of CPU this job got: 3%
Elapsed (wall clock) time (h:mm:ss or m:ss): 10:55.18
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 24712
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 3
Minor (reclaiming a frame) page faults: 2893
Voluntary context switches: 404552
Involuntary context switches: 17983
Swaps: 0
File system inputs: 728
File system outputs: 23612192
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

minikube image load docker.img

Command being timed: "minikube image load --daemon=false --remote=false --overwrite=true current-docker.img"
User time (seconds): 14.12
System time (seconds): 24.48
Percent of CPU this job got: 20%
Elapsed (wall clock) time (h:mm:ss or m:ss): 3:05.68
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 98460
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 275
Minor (reclaiming a frame) page faults: 19528
Voluntary context switches: 775099
Involuntary context switches: 27116
Swaps: 0
File system inputs: 50752
File system outputs: 432
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

SpecLad commented 7 months ago

/remove-lifecycle stale

k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

SpecLad commented 4 months ago

/remove-lifecycle stale

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

antl3x commented 1 month ago

/remove-lifecycle stale

kubernetes / minikube