[Bug]: "message":"failed to create a new hot reloader"

[Davidffry]

zot version

v2.0.3

Describe the bug

I have got this message : "message":"failed to create a new hot reloader"

To reproduce

1. Configuration

I change the access & secret key

# If mountConfig is true the configMap named $CHART_RELEASE-config is mounted
# on the pod's '/etc/zot' directory
mountConfig: true
# If mountConfig is true the chart creates the '$CHART_RELEASE-config', if it
# does not exist the user is in charge of managing it (as this file includes a
# sample file you have to add it empty to handle it externally) ... note that
# the service does not reload the configFiles once mounted, so you need to
# delete the pods to create new ones to use the new values.
configFiles:
  config.json: |-
    {
    "storage": {
        "rootDirectory": "/tmp/zot",
        "dedupe": false,
        "storageDriver": {
            "name": "s3",
            "region": "us-east-2",
            "bucket": "zot-storage",
            "secure": false,
            "skipverify": true,
            "accesskey": "abcd",
            "secretkey": "abcd",
            "regionendpoint": "http://abcd:9000"
        }
    },
      "http": { "address": "0.0.0.0", "port": "5000" },
      "log": { "level": "debug" }
    }

2. Client tool used

Kubernetes and the helm chart

4. Seen error

Error: too many open files
{"level":"error","error":"too many open files","goroutine":1,"caller":"zotregistry.dev/zot/pkg/cli/server/root.go:67","time":"2024-04-09T13:44:09.837674307Z","message":"failed to create a new hot reloader"}
Usage:
  zot serve <config> [flags]

Aliases:
  serve, serve

Flags:
  -h, --help   help for serve

Stream closed EOF for default/zot-76d754d4fb-bmsmq (zot)

Expected behavior

Works

Screenshots

No response

Additional context

Think that, I don't use the dedupe but in the doc not required. I think.

[rchincha]

sysctl fs.inotify.max_user_watches sysctl fs.inotify.max_user_instances sysctl fs.inotify.max_queued_events

@Davidffry can you print these values? maybe they are set too low?

[lesomnus]

I got the same issue. It seems that the problem is that the value of fs.inotify.max_user_instances was too low.

$ sysctl fs.inotify.max_user_instances
fs.inotify.max_user_instances = 128
$ sysctl -w fs.inotify.max_user_instances=1024
fs.inotify.max_user_instances = 1024

Even thoguh, I think 128 is already large enough. I asked to ChatGPT and how to get the number of instances, and the number is 56. Im not sure this number means "user_instances".

$ lsof | grep inotify | wc -l
56

P.S. Im running zot in the K3s which runs in the Docker container so: Docker Contaienr->K3s->containerd->zot