search

fluent / fluent-bit-docker-image

Docker image for Fluent Bit
https://hub.docker.com/r/fluent/fluent-bit/
Apache License 2.0
67 stars 75 forks source link

too many open files #41

Closed MrKervin closed 2 years ago

MrKervin commented 3 years ago

[docker@paas-bill-ccse-048 ~]$ kubectl exec -it account-accounting-grpc-7774d58977-p8fqn -n abm-prod -c accounting-fluent -- /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf Fluent Bit v1.4.4

[2021/08/27 03:45:37] [ info] [storage] version=1.0.3, initializing... [2021/08/27 03:45:37] [ info] [storage] in-memory [2021/08/27 03:45:37] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128 [2021/08/27 03:45:37] [ info] [engine] started (pid=81) [2021/08/27 03:45:37] [error] [plugins/in_tail/tail_fs.c:180 errno=24] Too many open files [2021/08/27 03:45:37] [error] Failed initialize input tail.0

The host /etc/security/limits.conf has been adjusted, but it didn't work. Don't know how to solve it。

Oznup commented 2 years ago

Hello, I have exactly the same issue installing fluent-bit with the helm chart. Did you manage to solve it ?

patrick-stephens commented 2 years ago

You've exhausted your host and process limits so there's not much that Fluent Bit can do here. It may be down to other workloads running on that same node or just trying to tail too many files. You could try a more recent version (1.4.4 is fairly old now) to see if any of the improvements help but the fundamental issue is down to exhausting the limits Fluent Bit is running with. The other option is reducing the number of files tailed in your configuration.

HattabbI4 commented 2 years ago

I have exactly the same issue installing fluent-bit with the helm chart.

I also have this problem in version Fluent Bit v1.9.5


[2022/10/31 12:14:45] [ info] [fluent bit] version=1.9.5, commit=9ec43447b6, pid=1
[2022/10/31 12:14:45] [ info] [storage] version=1.2.0, type=memory+filesystem, sync=normal, checksum=disabled, max_chunks_up=128
[2022/10/31 12:14:45] [ info] [storage] backlog input plugin: storage_backlog.1
[2022/10/31 12:14:45] [ info] [cmetrics] version=0.3.4
[2022/10/31 12:14:45] [error] [plugins/in_tail/tail_fs_inotify.c:304 errno=24] Too many open files
[2022/10/31 12:14:45] [error] Failed initialize input tail.0
[2022/10/31 12:14:45] [error] [lib] backend failed

did you manage to solve it?

edsiper commented 2 years ago

this is not a Fluent Bit issue, your parent process/environment allows just a low number of files to be open. Please increase your system limits associated with file descriptors and inotify max watchers.