Plugin not known after parsing fluent.conf file

[ButeForce]

I'm using fluentd image from docker hub and modified it to install some plugins including this one , the plugin gets installed successfully but when the container tries to start it cannot identify the mqtt pluging , I get the below error:

2020-07-29 22:54:41 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf" 2020-07-29 22:54:41 +0000 [info]: gem 'fluent-plugin-kafka' version '0.13.0' 2020-07-29 22:54:41 +0000 [info]: gem 'fluent-plugin-pgjson' version '1.0.0' 2020-07-29 22:54:41 +0000 [info]: gem 'fluent-plugin-sql' version '2.0.0' 2020-07-29 22:54:41 +0000 [info]: gem 'fluentd' version '1.9.3' 2020-07-29 22:54:41 +0000 [info]: brokers has been set: ["my-kafka:9092"] 2020-07-29 22:54:41 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Unknown input plugin 'mqtt'. Run 'gem search -rd fluent-plugin' to find plugins"

Any idea what could be the cause for this error ?

My fluent.conf file lokks as following:

<source>
      @type http
      port 9880
      bind 0.0.0.0
</source>

<source>
    @type mqtt
    host mqtt-server-emqx
    port 1883
    <security>
      username hamdi@rits.com
      password P@ssw0rd
    </security>
    <parse>
       @type json
    </parse>
</source>

<match *.*>
    @type kafka2
    # list of seed brokers
    brokers my-kafka:9092
    #use_event_time true
    # buffer settings
    <buffer topic>
        @type file
         path /var/log/td-agent/buffer/td
         flush_interval 1s
    </buffer>
    # data type settings
    <format>
       @type json
    </format>
    # topic settings
    #topic_key temp_1
    default_topic Aqua
    # producer settings
    required_acks -1
    compression_codec gzip
 </match>

[toyokazu]

Could you check the following command by the user who starts up fluentd inside the docker instance?

gem list fluent-plugin

Furthermore, could you also check that the plugin can be loaded by your default ruby?

irb

gem 'fluent-plugin-mqtt-io'

Best

[ButeForce]

Dear toyokazu,

Actually I'm using kubernetes to deploy the docker, and the container fails once it is deployed and I cannot exec into it , but using the Docker file it was set to use fluent user and I tried to make it root also with same result the Docker file I use to build the docker as below:

# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /Dockerfile.template.erb

FROM ruby:2.6-slim-buster
LABEL maintainer "Fluentd developers <fluentd@googlegroups.com>"
LABEL Description="Fluentd docker image" Vendor="Fluent Organization" Version="1.9.3"
ENV TINI_VERSION=0.18.0

# Do not split this into multiple RUN!
# Docker creates a layer for every RUN-Statement
# therefore an 'apt-get purge' has no effect
RUN apt-get update 
RUN apt-get install libpq-dev -y --install-recommends
RUN apt-get install -y --no-install-recommends \
            ca-certificates \
 && buildDeps=" \
      make gcc g++ libc-dev \
      wget bzip2 gnupg dirmngr \
    " \
 && apt-get install -y --no-install-recommends $buildDeps \
 && echo 'gem: --no-document' >> /etc/gemrc \
 && gem install oj -v 3.8.1 \
 && gem install json -v 2.3.0 \
 && gem install async-http -v 0.50.0 \
 && gem install ext_monitor -v 0.1.2 \
 && gem install fluentd -v 1.9.3 \
 && gem install fluent-plugin-mqtt-io -v 0.4.4\
 && gem install fluent-plugin-kafka \
 && gem install pg \
 && gem install fluent-plugin-sql --no-document \
 && gem install pg --no-document \
 && gem uninstall tzinfo -v 2.0.2 \
 && dpkgArch="$(dpkg --print-architecture | awk -F- '{ print $NF }')" \
 && wget -O /usr/local/bin/tini "https://github.com/krallin/tini/releases/download/v$TINI_VERSION/tini-$dpkgArch" \
 && wget -O /usr/local/bin/tini.asc "https://github.com/krallin/tini/releases/download/v$TINI_VERSION/tini-$dpkgArch.asc" \
 && export GNUPGHOME="$(mktemp -d)" \
 && gpg --batch --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 6380DC428747F6C393FEACA59A84159D7001A4E5 \
 && gpg --batch --verify /usr/local/bin/tini.asc /usr/local/bin/tini \
 && rm -r /usr/local/bin/tini.asc \
 && chmod +x /usr/local/bin/tini \
 && tini -h \
 && wget -O /tmp/jemalloc-4.5.0.tar.bz2 https://github.com/jemalloc/jemalloc/releases/download/4.5.0/jemalloc-4.5.0.tar.bz2 \
 && cd /tmp && tar -xjf jemalloc-4.5.0.tar.bz2 && cd jemalloc-4.5.0/ \
 && ./configure && make \
 && mv lib/libjemalloc.so.2 /usr/lib \
 && apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

RUN groupadd -r fluent && useradd -r -g fluent fluent \
    # for log storage (maybe shared with host)
    && mkdir -p /fluentd/log \
    # configuration/plugins path (default: copied from .)
    && mkdir -p /fluentd/etc /fluentd/plugins \
    && chown -R fluent /fluentd && chgrp -R fluent /fluentd

COPY fluent.conf /fluentd/etc/
COPY entrypoint.sh /bin/

ENV FLUENTD_CONF="fluent.conf"

ENV LD_PRELOAD="/usr/lib/libjemalloc.so.2"
EXPOSE 24224 5140

USER root
ENTRYPOINT ["tini",  "--", "/bin/entrypoint.sh"]
CMD ["fluentd"]

Also while building the image Im able to see the plugin installed successfully both mqtt and fluent-plugin-mqtt-io

[ButeForce]

it seems something not very clear corrupted the fluentd image , I deleted all the images and rebuild from scratch , it could work again

[toyokazu]

Hmm. If you find the root cause, please share it here or somewhere related to the docker image developer community. It will help people having the same trouble.

Anyway, I'm glad to hear that it was resolved. I'd like to close this issue.