search

toyokazu / fluent-plugin-mqtt-io

Fluent plugin for MQTT Input/Output
Apache License 2.0
17 stars 12 forks source link

Unknown output plugin 'mqtt' when used in <store> with @type copy #23

Open shane-wareham opened 3 years ago

shane-wareham commented 3 years ago

I'm trying to use the fluent-plugin-mqtt-io to send logging data to a mqtt broker with fluentd. There are multiple outputs being used, and, I wanted to add a to the copy list. When I add the section and restart fluentd, it fails to restart claiming it can't find the 'mqtt' plugin.

This is running on a VM with Windows Server 2016, 32 GB and 8-cores. The td-agent.conf calls two separate conf files.

Include the other config file that contains all the source configuration

@include td-agent.stage.pssturnpike.conf @include td-agent.stage.leidmanagement.conf

The section was added to td-agent.stage.pssturnpike.conf. and looked like this (IP address has been redacted):

@type mqtt @type json add_newline false host xx.xxx.xxx.xxx port 11883

When fluentd was restarted, the log stopped at this message.

2021-01-05 12:26:12 -0500 [error]: config error file="C:/opt/td-agent/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Unknown output plugin 'mqtt'. Run 'gem search -rd fluent-plugin' to find plugins"

The fluent plugins are as follows:

C:\opt\td-agent>gem list fluent-plugin

LOCAL GEMS

fluent-plugin-elasticsearch (4.0.7) fluent-plugin-kafka (0.7.9) fluent-plugin-mqtt-io (0.4.4) fluent-plugin-multi-format-parser (1.0.0) fluent-plugin-parser-winevt_xml (0.2.2) fluent-plugin-prometheus (1.7.3) fluent-plugin-prometheus_pushgateway (0.0.2) fluent-plugin-record-modifier (2.1.0) fluent-plugin-rewrite-tag-filter (2.2.0) fluent-plugin-s3 (1.3.1) fluent-plugin-secure-forward (0.4.5) fluent-plugin-td (1.1.0) fluent-plugin-td-monitoring (0.2.4) fluent-plugin-webhdfs (1.2.4) fluent-plugin-windows-eventlog (0.5.4)

And the mqtt plugins are:

C:\opt\td-agent>gem list mqtt

LOCAL GEMS

fluent-plugin-mqtt-io (0.4.4) mqtt (0.5.0)

Checking with Ruby, it appear to load properly:

irb(main):001:0> gem 'fluent-plugin-mqtt-io' => true irb(main):002:0> exit

When the section for mqtt is included, and, fluentd is restarted, it stops on the mqtt plugin. (host names and IP addresses have been removed)

2021-01-05 12:26:11 -0500 [info]: parsing config file is succeeded path="C:/opt/td-agent/etc/td-agent/td-agent.conf" 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.7' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-kafka' version '0.7.9' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-mqtt-io' version '0.4.4' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-parser-winevt_xml' version '0.2.2' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-prometheus' version '1.7.3' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.0.2' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-s3' version '1.3.1' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-secure-forward' version '0.4.5' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-td' version '1.1.0' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-webhdfs' version '1.2.4' 2021-01-05 12:26:11 -0500 [info]: gem 'fluent-plugin-windows-eventlog' version '0.5.4' 2021-01-05 12:26:11 -0500 [info]: gem 'fluentd' version '1.10.2' 2021-01-05 12:26:11 -0500 [warn]: [out_pss_forward] 'heartbeat_type tcp' is deprecated. use 'transport' instead. 2021-01-05 12:26:11 -0500 [warn]: [out_pss_forward] TLS transport is configured in insecure way 2021-01-05 12:26:11 -0500 [info]: [out_pss_forward] adding forwarding server '' host="" port=24224 weight=60 plugin_id="out_pss_forward" 2021-01-05 12:26:12 -0500 [info]: [out_pss_kafka] brokers has been set: ["", "", ""] 2021-01-05 12:26:12 -0500 [warn]: [out_pss_csi_forward] 'heartbeat_type tcp' is deprecated. use 'transport' instead. 2021-01-05 12:26:12 -0500 [info]: [out_pss_csi_forward] adding forwarding server '' host="" port=7934 weight=60 plugin_id="out_pss_csi_forward" 2021-01-05 12:26:12 -0500 [error]: config error file="C:/opt/td-agent/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Unknown output plugin 'mqtt'. Run 'gem search -rd fluent-plugin' to find plugins"

When the for mqtt is removed, the fluent process continues and looks like this (host names and IP addresses have been removed):

2021-01-05 12:28:13 -0500 [info]: parsing config file is succeeded path="C:/opt/td-agent/etc/td-agent/td-agent.conf" 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.7' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-kafka' version '0.7.9' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-mqtt-io' version '0.4.4' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-parser-winevt_xml' version '0.2.2' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-prometheus' version '1.7.3' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.0.2' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-s3' version '1.3.1' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-secure-forward' version '0.4.5' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-td' version '1.1.0' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-webhdfs' version '1.2.4' 2021-01-05 12:28:13 -0500 [info]: gem 'fluent-plugin-windows-eventlog' version '0.5.4' 2021-01-05 12:28:13 -0500 [info]: gem 'fluentd' version '1.10.2' 2021-01-05 12:28:13 -0500 [warn]: [out_pss_forward] 'heartbeat_type tcp' is deprecated. use 'transport' instead. 2021-01-05 12:28:13 -0500 [warn]: [out_pss_forward] TLS transport is configured in insecure way 2021-01-05 12:28:13 -0500 [info]: [out_pss_forward] adding forwarding server '' host="" port=24224 weight=60 plugin_id="out_pss_forward" 2021-01-05 12:28:13 -0500 [info]: [out_pss_kafka] brokers has been set: ["", "", ""] 2021-01-05 12:28:13 -0500 [warn]: [out_pss_csi_forward] 'heartbeat_type tcp' is deprecated. use 'transport' instead. 2021-01-05 12:28:13 -0500 [info]: [out_pss_csi_forward] adding forwarding server '' host="" port=7934 weight=60 plugin_id="out_pss_csi_forward" 2021-01-05 12:28:13 -0500 [warn]: [out_leid_forward] 'heartbeat_type tcp' is deprecated. use 'transport' instead. 2021-01-05 12:28:13 -0500 [warn]: [out_leid_forward] TLS transport is configured in insecure way 2021-01-05 12:28:13 -0500 [info]: [out_leid_forward] adding forwarding server '' host="" port=24224 weight=60 plugin_id="out_leid_forward" 2021-01-05 12:28:13 -0500 [info]: [out_leid_kafka] brokers has been set: ["", "", ""] 2021-01-05 12:28:13 -0500 [info]: using configuration file:

The other three sections are loaded and run correctly with the same file configuration, just the section for mqtt was removed.

I can send full configuration files if needed.

toyokazu commented 3 years ago

Are you sure to execute "gem" embedded to the td-agent?

https://docs.fluentd.org/deployment/plugin-management

You need to use the embedded gem if you want to use the plugin with td-agent.

Best

shane-wareham commented 3 years ago

[cid:image001.png@01D6E408.E5CCE4D0]

From the td-agent command prompt…

C:\fluent-gems>fluent-gem install --force --local *.gem Successfully installed fluent-plugin-kafka-0.7.9 Parsing documentation for fluent-plugin-kafka-0.7.9 Done installing documentation for fluent-plugin-kafka after 0 seconds Successfully installed fluent-plugin-mqtt-io-0.4.4 Parsing documentation for fluent-plugin-mqtt-io-0.4.4 Done installing documentation for fluent-plugin-mqtt-io after 0 seconds Successfully installed fluent-plugin-secure-forward-0.4.5 Parsing documentation for fluent-plugin-secure-forward-0.4.5 Done installing documentation for fluent-plugin-secure-forward after 0 seconds Successfully installed mqtt-0.5.0 Parsing documentation for mqtt-0.5.0 Done installing documentation for mqtt after 0 seconds Successfully installed ruby-kafka-0.5.5 Parsing documentation for ruby-kafka-0.5.5 Done installing documentation for ruby-kafka after 1 seconds 5 gems installed

And I still see the error…

2021-01-06 09:43:54 -0500 [error]: config error file="C:/opt/td-agent/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Unknown output plugin 'mqtt'. Run 'gem search -rd fluent-plugin' to find plugins"

I’ve taken this over from a different group, and, this is how they have setup the gem files.

Thanks, Shane Wareham Principal Developer - Host Systems Application Development | Information Technology I will be out of the office from Dec 21 through Dec 27

From: Toyokazu Akiyama notifications@github.com Sent: Tuesday, January 5, 2021 7:50 PM To: toyokazu/fluent-plugin-mqtt-io fluent-plugin-mqtt-io@noreply.github.com Cc: Wareham, Shane Shane.Wareham@united.com; Author author@noreply.github.com Subject: Re: [toyokazu/fluent-plugin-mqtt-io] Unknown output plugin 'mqtt' when used in with @type copy (#23)

Are you sure to execute "gem" embedded to the td-agent?

https://docs.fluentd.org/deployment/plugin-managementhttps://urldefense.com/v3/__https:/docs.fluentd.org/deployment/plugin-management__;!!HYthtcT8!IfaHF9TGmTy2hCF_gZctuvC23oHNNtbMz8jQ8CqLUg7iH3zgjd4RkU_AqxXNWRVfHQ$

You need to use the embedded gem if you want to use the plugin with td-agent.

Best

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/toyokazu/fluent-plugin-mqtt-io/issues/23*issuecomment-755023469__;Iw!!HYthtcT8!IfaHF9TGmTy2hCF_gZctuvC23oHNNtbMz8jQ8CqLUg7iH3zgjd4RkU_AqxVzUHEqCA$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AMGLW6WBDLUFEAPVBUQPV63SYO6VTANCNFSM4VWBY5JA__;!!HYthtcT8!IfaHF9TGmTy2hCF_gZctuvC23oHNNtbMz8jQ8CqLUg7iH3zgjd4RkU_AqxU62s8Ckg$.

toyokazu commented 3 years ago

Are you really using fluentd NOT td-agent? The following page said to use td-agent-gem for td-agent...

https://docs.fluentd.org/deployment/plugin-management

Anyway, I could not reproduce the error in my environment with the copy directive. I'm sorry that I can't be of any help.

vi out_mqtt_example.conf 

<source>
  @type mqtt
  host localhost
  port 1883
  topic app/test
  <parse>
    @type none
  </parse>
</source>

<match app.**>
  @type copy
  <store>
    @type mqtt
    host localhost
    port 1883
    topic_rewrite_pattern '^([\w\/]+)$'
    topic_rewrite_replacement '\1/rewritten1'
  </store>
  <store>
    @type elasticsearch
    host localhost
    port 9200
    index_name my-index-01
  </store>
  <store>
    @type mqtt
    host localhost
    port 1883
    topic_rewrite_pattern '^([\w\/]+)$'
    topic_rewrite_replacement '\1/rewritten2'
  </store>
</match>
brew services start mosquito
fluentd -v -c out_mqtt_example.conf

mosquitto_sub -t app/test/rewritten1

mosquitto_sub -t app/test/rewritten2

mosquitto_pub -t app/test -m hogefoo

With the above configuration, two subscribers correctly receive the message and it is also stored into elasticsearch.

shane-wareham commented 3 years ago

Td-agent-gem is a bash script that isn’t supported on Windows (I could convert it to a .bat file). The script simply calls fleunt-gem

!/bin/sh

export GEM_HOME="C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/" export GEM_PATH="C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/" C:/opt/td-agent/embedded/bin/fluent-gem "$@"

So, I called fluent-gem directly.

I’ll look through your configuration to see if there is something I’ve left out.

Thanks for your help, Shane Wareham Principal Developer - Host Systems Application Development | Information Technology I will be out of the office from Dec 21 through Dec 27

From: Toyokazu Akiyama notifications@github.com Sent: Wednesday, January 6, 2021 11:15 AM To: toyokazu/fluent-plugin-mqtt-io fluent-plugin-mqtt-io@noreply.github.com Cc: Wareham, Shane Shane.Wareham@united.com; Author author@noreply.github.com Subject: Re: [toyokazu/fluent-plugin-mqtt-io] Unknown output plugin 'mqtt' when used in with @type copy (#23)

Are you really using fluentd NOT td-agent? The following page said to use td-agent-gem for td-agent...

https://docs.fluentd.org/deployment/plugin-managementhttps://urldefense.com/v3/__https:/docs.fluentd.org/deployment/plugin-management__;!!HYthtcT8!Krmy09Apb6VuJC26iqiQ9hVYYY5pkBmzZB9Y5zTpjm7SoR1urM3npTelCmO5dzIgDQ$

Anyway, I could not reproduce the error in my environment with the copy directive. I'm sorry that I can't be of any help.

vi out_mqtt_example.conf

@type mqtt

host localhost

port 1883

topic app/test

@type none

<match app.**>

@type copy

@type mqtt host localhost port 1883 topic_rewrite_pattern '^([\w\/]+)$' topic_rewrite_replacement '\1/rewritten1' @type elasticsearch host localhost port 9200 index_name my-index-01 @type mqtt host localhost port 1883 topic_rewrite_pattern '^([\w\/]+)$' topic_rewrite_replacement '\1/rewritten2'

brew services start mosquito

fluentd -v -c out_mqtt_example.conf

mosquitto_sub -t app/test/rewritten1

mosquitto_sub -t app/test/rewritten2

mosquitto_pub -t app/test -m hogefoo

With the above configuration, two subscribers correctly receive the message and it is also stored into elasticsearch.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/toyokazu/fluent-plugin-mqtt-io/issues/23*issuecomment-755436121__;Iw!!HYthtcT8!Krmy09Apb6VuJC26iqiQ9hVYYY5pkBmzZB9Y5zTpjm7SoR1urM3npTelCmNSRrAm4Q$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AMGLW6XA3E4V6VH24N5PVL3SYSLBPANCNFSM4VWBY5JA__;!!HYthtcT8!Krmy09Apb6VuJC26iqiQ9hVYYY5pkBmzZB9Y5zTpjm7SoR1urM3npTelCmND3aDRFA$.