"too many open files" error when install kubevirt in cnstack

[yehaifeng]

When i installed kubevirt in cnstack, but has a error about virt-handler. The err logs as follows:

{"component":"virt-handler","hostname":"yhf02","level":"info","pos":"virt-handler.go:195","timestamp":"2023-05-04T06:52:07.918682Z"}
W0504 06:52:07.922104   92191 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:455","timestamp":"2023-05-04T06:52:07.939817Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:464","timestamp":"2023-05-04T06:52:07.939924Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:455","timestamp":"2023-05-04T06:52:07.939945Z"}
{"component":"virt-handler","level":"info","msg":"CPU features of a minimum baseline CPU model: map[apic:true clflush:true cmov:truecx16:true cx8:true de:true fpu:true fxsr:true lahf_lm:true lm:true mca:true mce:true mmx:true msr:true mtrr:true nx:true pae:true pat:true pge:true pni:true pse:true pse36:true sep:true sse:true sse2:true sse4.1:true ssse3:true syscall:true tsc:true]","pos":"cpu_plugin.go:96","timestamp":"2023-05-04T06:52:08.020292Z"}
{"component":"virt-handler","level":"warning","msg":"host model mode is expected to contain only one model","pos":"cpu_plugin.go:103","timestamp":"2023-05-04T06:52:08.020344Z"}
{"component":"virt-handler","level":"info","msg":"node-labeller is running","pos":"node_labeller.go:94","timestamp":"2023-05-04T06:52:08.021374Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:455","timestamp":"2023-05-04T06:52:08.021708Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:464","timestamp":"2023-05-04T06:52:08.021810Z"}
{"component":"virt-handler","level":"info","msg":"Starting domain stats collector: node name=yhf02","pos":"prometheus.go:472","timestamp":"2023-05-04T06:52:08.022603Z"}
{"component":"virt-handler","level":"info","msg":"Starting Guest OS Info collector: node name=yhf02","pos":"prometheus_guest_info.go:65","timestamp":"2023-05-04T06:52:08.022653Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer vmiInformer-sources","pos":"virtinformers.go:306","timestamp":"2023-05-04T06:52:08.022775Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer vmiInformer-targets","pos":"virtinformers.go:306","timestamp":"2023-05-04T06:52:08.022832Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer extensionsKubeVirtCAConfigMapInformer","pos":"virtinformers.go:306","timestamp":"2023-05-04T06:52:08.022876Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer CRDInformer","pos":"virtinformers.go:306","timestamp":"2023-05-04T06:52:08.022897Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer kubeVirtInformer","pos":"virtinformers.go:306","timestamp":"2023-05-04T06:52:08.022912Z"}
{"component":"virt-handler","level":"info","msg":"metrics: max concurrent requests=3","pos":"virt-handler.go:479","timestamp":"2023-05-04T06:52:08.023746Z"}
{"component":"virt-handler","level":"fatal","msg":"Failed to create an inotify watcher","pos":"cert-manager.go:105","reason":"too many open files","timestamp":"2023-05-04T06:52:08.024081Z"}
panic: Failed to create an inotify watcher

goroutine 104 [running]:
kubevirt.io/client-go/log.FilteredLogger.Critical({{0x25c9b00, 0xc003ba95e0}, {0x20e67c2, 0xc}, 0x0, 0x0, 0x2, 0x2, {0x25ce760, 0x3ab6d60}}, ...)
    staging/src/kubevirt.io/client-go/log/log.go:338 +0x196
kubevirt.io/kubevirt/pkg/certificates/bootstrap.(*FileCertificateManager).Start(0xc0004aa550)
    pkg/certificates/bootstrap/cert-manager.go:105 +0x1a5
created by main.(*virtHandlerApp).Run
    cmd/virt-handler/virt-handler.go:368 +0x1bcf

[lxs137]

Please increase the amount of inotify watches allowed per node via `/proc/sys/fs/inotify/max_user_watches'.

Temporary fix is to sudo sysctl fs.inotify.max_user_watches=524288 Permanent fix is to echo "fs.inotify.max_user_watches=524288" >> /etc/sysctl.conf

[yehaifeng]

Please increase the amount of inotify watches allowed per node via `/proc/sys/fs/inotify/max_user_watches'.

Temporary fix is to sudo sysctl fs.inotify.max_user_watches=524288 Permanent fix is to echo "fs.inotify.max_user_watches=524288" >> /etc/sysctl.conf

Thanks！

Ubuntu 22.04 has this problem, but used CentOS 7.9 and Ubuntu 20.04 is ok!