Too many open files when spinning up multiple kind clusters

tylerauerbeck commented 3 months ago

Description

I'm attempting to run a POC locally that requires multiple kind clusters to demonstrate. The first three come up without issue, however the fourth has problems coming up. To be clear, I don't think this has anything to do with kind specifically, but more that I'm bringing up a number of containers that are probably a bit more resource intensive which require adjusting various settings.

Specifically in my case I see the following in the logs of the fourth container

Failed to create inotify object: Too many open files
         Starting systemd-remount-f…nt Root and Kernel File Systems...
Failed to create inotify object: Too many open files
[  OK  ] Finished kmod-static-nodes…reate List of Static Device Nodes.
Failed to create inotify object: Too many open files

Is there a way to easily update the settings to allow for the necessary inotify params to be tuned? Is this easily done as part of a config file vs having to manually do this within the VM itself?

Version

> colima version && limactl --version && qemu-img --version
colima version 0.6.8
git commit: 9b0809d0ed9ad3ff1e57c405f27324e6298ca04f

runtime: docker
arch: aarch64
client: v25.0.4
server: v24.0.9
limactl version 0.20.1
qemu-img version 8.2.1
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers

Operating System

[ ] macOS Intel <= 13 (Ventura)
[ ] macOS Intel >= 14 (Sonoma)
[ ] Apple Silicon <= 13 (Ventura)
[X] Apple Silicon >= 14 (Sonoma)
[ ] Linux

Output of `colima status`

> colima status
INFO[0000] colima is running using QEMU
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: sshfs
INFO[0000] socket: unix:///Users/tyler/.colima/default/docker.sock

Reproduction Steps

colima start --cpu 6 --memory 24
kind create cluster --name cluster-1
kind create cluster --name cluster-2
kind create cluster --name cluster-3
kind create cluster --name cluster-4 (will fail with the above logs)

Expected behaviour

Four kind clusters can come up without issue within colima

Additional context

I haven't done anything special to configure colima. This is about as clean of an install as can be as I just installed this via homebrew

bayasdev commented 3 months ago

@tylerauerbeck you need to increase the file handler limit of the underlying Linux distro (Ubuntu) running on Colima

Edit /etc/sysctl.conf

fs.inotify.max_user_watches = 1048576
fs.inotify.max_user_instances = 512

Run sudo systctl -p or reboot the Colima VM

j-sauer commented 2 months ago

@bayasdev How do you edit this file? I tried to connect to the vm via colima ssh but I only see the file system of my Mac. Started Colima with colima start --arch x86_64 --memory 16 --cpu 6 --kubernetes --vm-type vz -s

bayasdev commented 2 months ago

@bayasdev How do you edit this file?

I tried to connect to the vm via colima ssh but I only see the file system of my Mac.

Started Colima with

colima start --arch x86_64 --memory 16 --cpu 6 --kubernetes --vm-type vz -s

To connecting to the VM you need to type

colima ssh

abiosoft / colima