fluent / fluentd-docker-image

Docker image for Fluentd
https://hub.docker.com/r/fluent/fluentd/
Apache License 2.0
464 stars 351 forks source link

[alpine images] Coolio::StatWatcher fallbacks to polling instead of using inotify #113

Open adw1n opened 6 years ago

adw1n commented 6 years ago

Description

When I use the fluent/fluentd:v1.1.0 image the in_tail plugin fallbacks to polling file every 5 seconds instead of using inotify. This causes loss in timestamp precision. fluent/fluentd:v1.1.0-debian image handles watching file correctly and does not fallback to polling.

Steps to reproduce

  1. put Dockerfile, fluent.conf and write_logs.sh in one directory Dockerfile
    
    FROM fluent/fluentd:v1.1.0

COPY ./fluent.conf /fluentd/etc/fluent.conf COPY ./write_logs.sh /usr/local/bin/write-logs RUN chmod 755 /usr/local/bin/write-logs

RUN find . -name in_tail.rb | xargs sed -i '/line.chomp!/a \ \ \ \ \ \ \ \ puts Time.now.to_datetime.iso8601(3)'

fluent.conf

@type tail path /tmp/test.log pos_file /tmp/test.log.pos tag test path_key tailed_test keep_time_key true format none enable_watch_timer false

write_logs.sh

!/bin/sh

counter=0 while true; do sleep 0.01 echo $counter >> /tmp/test.log counter=expr $counter + 1 done


2. run `docker build -t inotify_problem . && docker run --name inotify_problem -it inotify_problem`
3. run `docker exec -it inotify_problem write-logs`

As you can see, logs are being polled with 5 sec frequency (`convert_line_to_event` function is executed every ~5 sec).

Now change `fluent/fluentd:v1.1.0` to `fluent/fluentd:v1.1.0-debian` in the Dockerfile and run:
1. `docker stop inotify_problem; docker rm inotify_problem; docker rmi inotify_problem`
2. `docker build -t inotify_problem . && docker run --name inotify_problem -it inotify_problem`
3. `docker exec -it inotify_problem write-logs`

Observe that `convert_line_to_event` function is executed every ~0.01 sec.
github-actions[bot] commented 2 months ago

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days