Encountering `failed to create fsnotify watcher: too many open files` getting logs

[imle]

Bug Report

Description

When streaming logs from a service, I see failed to create fsnotify watcher: too many open files.

Logs

Can add here, not really sure where to look.

Environment

Proxmox cluster with 3 hosts. Ceph running through Proxmox. Exposed to cluster through Rook.

• Talos version:

  Client:
  Tag:         v0.14.3
  SHA:         73f4efd7
  Built:
  Go version:  go1.17.7
  OS/Arch:     darwin/amd64
  Server:
  NODE:        10.0.50.102
  Tag:         v0.14.1
  SHA:         7182bcf2
  Built:
  Go version:  go1.17.5
  OS/Arch:     linux/amd64
  Enabled:

• Kubernetes version:

  Client Version: v1.22.4
  Server Version: v1.23.1

• Platform: Proxmox

[githubcdr]

You could add this to sysctl to increase the open files limit;

fs.inotify.max_user_instances: "512"

It has been included in the latest Talos release, https://github.com/talos-systems/talos/commit/cbc9610be

[imle]

Thanks! I'll go ahead and update Talos as this is a dev cluster anyway. Maybe I am missing this while searching the docs, is there a way to read its current value?

[frezbo]

talosctl get kernelparams