Closed vasanthaneppali closed 2 weeks ago
I think a default fluentd image does not include elasticsearch plugin and you have to add it to a custom image.
@vasanthaneppali sample here
FROM fluent/fluentd:v1.6-debian-1
USER root
RUN apt update -y
RUN apt install -y ubuntu-dev-tools
RUN gem install elasticsearch -v 7.17.1 --no-document
RUN gem install fluent-plugin-elasticsearch -v 5.0.3 --no-document
RUN gem install fluent-plugin-prometheus -v 2.0.1 --no-document
USER fluent
@junejie This actually will not work properly, as the plugin gems will not be readable/executable by the fluent user in the correct way. I just tried this exact way, and this resulted in errors like this:
<internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- /usr/local/bundle/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions (LoadError)
from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/registry.rb:103:in `block in search'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/registry.rb:100:in `each'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/registry.rb:100:in `search'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/registry.rb:44:in `lookup'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/plugin.rb:169:in `new_impl'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/plugin.rb:114:in `new_output'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/agent.rb:130:in `add_match'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/agent.rb:74:in `block in configure'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/agent.rb:64:in `each'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/agent.rb:64:in `configure'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/root_agent.rb:149:in `configure'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/engine.rb:105:in `configure'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/engine.rb:80:in `run_configure'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/supervisor.rb:708:in `run_supervisor'
from /usr/local/bundle/gems/fluentd-1.15.1/lib/fluent/command/fluentd.rb:366:in `<top (required)>'
from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from /usr/local/bundle/gems/fluentd-1.15.1/bin/fluentd:15:in `<top (required)>'
from /usr/local/bundle/bin/fluentd:25:in `load'
from /usr/local/bundle/bin/fluentd:25:in `<main>'
EDIT: Reason being the read permissions of the gem files:
fluent@07e77483ed31:/usr/local/bundle/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin$ ls -lia
total 32
3849008 drwxr-xr-x 2 root root 4096 Aug 24 15:33 .
3849007 drwxr-xr-x 3 root root 4096 Aug 24 15:33 ..
3849010 -rw-r----- 1 root root 12916 Aug 24 15:33 exception_detector.rb
3849011 -rw-r----- 1 root root 4614 Aug 24 15:33 out_detect_exceptions.rb
EDIT 2: My suggestion is to either add documentation that the files mode/ownership also needs to be updated. Or that --user-install
should be supported (if it is not already)
It may depend on your case, daemonset image may be useful for you. (they provides image for elasticsearch 7) https://github.com/fluent/fluentd-kubernetes-daemonset
It is out of scope in this repository, so I'll close this issue.
i am trying to copy a volume and not able to make the container up
/usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/config/types.rb:96: warning: regular expression has ']' without escape 2022-05-07 16:38:16 +0530 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::NotFoundPluginError error="Unknown output plugin 'elasticsearch'. Run 'gem search -rd fluent-plugin' to find plugins"
command docker run -itd --user root --restart always -v fluentd-volume:/fluentd/etc --name fluentd -p "24225:24225" fluent/fluentd:v1.14-debian-1