search

toyokazu / fluent-plugin-mqtt-io

Fluent plugin for MQTT Input/Output
Apache License 2.0
17 stars 12 forks source link

Error caused when load module #18

Closed otokunaga2 closed 4 years ago

otokunaga2 commented 4 years ago

Hello. I would like to report that I caused an error both input & output plugin in my environment.

I suspect that original Fluentd is currently major updated. Hence, the plugin's structure seems a little changed and which may influenced. The detailed information is as follows. To cope the issue I create the PR, so could you check?

My environment.

Fluentd config(output plugin)

<source>
  @type forward
</source>

<match topic.**>
  @type mqtt
  host 127.0.0.1
  port 1883
  <format>
    @type json
    add_newline false
  </format>
  topic_rewrite_pattern '^([\w\/]+)$'
  topic_rewrite_replacement '\1/rewritten'
</match>

<match **>
  @type stdout
</match>

Stacktraces


        13: from /var/lib/gems/2.5.0/gems/fluentd-1.10.4/lib/fluent/root_agent.rb:146:in `configure'
        12: from /var/lib/gems/2.5.0/gems/fluentd-1.10.4/lib/fluent/agent.rb:64:in `configure'
        11: from /var/lib/gems/2.5.0/gems/fluentd-1.10.4/lib/fluent/agent.rb:64:in `each'
        10: from /var/lib/gems/2.5.0/gems/fluentd-1.10.4/lib/fluent/agent.rb:74:in `block in configure'
         9: from /var/lib/gems/2.5.0/gems/fluentd-1.10.4/lib/fluent/agent.rb:130:in `add_match'
         8: from /var/lib/gems/2.5.0/gems/fluentd-1.10.4/lib/fluent/plugin.rb:109:in `new_output'
         7: from /var/lib/gems/2.5.0/gems/fluentd-1.10.4/lib/fluent/plugin.rb:155:in `new_impl'
         6: from /var/lib/gems/2.5.0/gems/fluentd-1.10.4/lib/fluent/registry.rb:44:in `lookup'
         5: from /var/lib/gems/2.5.0/gems/fluentd-1.10.4/lib/fluent/registry.rb:68:in `search'
         4: from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
         3: from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
         2: from /home/otokunaga2/work/fluent-plugin-mqtt-io/lib/fluent/plugin/out_mqtt.rb:4:in `<top (required)>'
         1: from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
/usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- fluent/plugin/mqtt_proxy (LoadError)```
toyokazu commented 4 years ago

Hi,

I've also tested in the following environment without any errors. In your error message, the following error seems to indicate the root cause.

/home/otokunaga2/work/fluent-plugin-mqtt-io/lib/fluent/plugin/out_mqtt.rb:4:in `<top (required)>'

You are trying the plugin without installing it by gem but just downloading it to your local folder. It requires you to set $LOAD_PATH. Please confirm your environment again.

Best regards

Ubuntu 18.04.4
rruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]
fluent-plugin-mqtt-io (0.4.4)
fluentd (1.10.4)

sudo apt install ruby ruby-dev mosquitto
sudo gem install fluent-plugin-mqtt-io

mkdir fluentd
cd fluentd
git clone https://github.com/OpenVPN/easy-rsa.git
cd easy-rsa/easyrsa3
./easyrsa init-pki
./easyrsa build-ca
./easyrsa gen-req localhost nopass
./easyrsa sign-req server localhost
./easyrsa gen-req client nopass
./easyrsa sign-req client client
cd ../../
vi fluent-in_mqtt-tls.conf 
---
<source>
  @type mqtt
  host localhost
  port 8883
  topic test
  <parse>
    @type none
  </parse>
  <security>
    use_tls true
    <tls>
      ca_file /home/ubuntu/fluentd/easy-rsa/easyrsa3/pki/ca.crt
      cert_file /home/ubuntu/fluentd/easy-rsa/easyrsa3/pki/issued/client.crt
      key_file /home/ubuntu/fluentd/easy-rsa/easyrsa3/pki/private/client.key
    </tls>
  </security>
  @label @test
</source>

<label @test>
  <match **>
    @type stdout
  </match>
</label>
---

vi fluent-out_mqtt-tls.conf 
---
<source>
  @type mqtt
  host localhost
  port 8883
  topic test
  <parse>
    @type none
  </parse>
  <security>
    use_tls true
    <tls>
      ca_file /home/ubuntu/fluentd/easy-rsa/easyrsa3/pki/ca.crt
      cert_file /home/ubuntu/fluentd/easy-rsa/easyrsa3/pki/issued/client.crt
      key_file /home/ubuntu/fluentd/easy-rsa/easyrsa3/pki/private/client.key
    </tls>
  </security>
  @label @test
</source>

<label @test>
  <match **>
    @type mqtt
    host localhost
    port 8883
    topic_rewrite_pattern '^([\w\/]+)$'
    topic_rewrite_replacement '\1/rewritten'
    <security>
      use_tls true
      <tls>
      ca_file /home/ubuntu/fluentd/easy-rsa/easyrsa3/pki/ca.crt
      cert_file /home/ubuntu/fluentd/easy-rsa/easyrsa3/pki/issued/client.crt
      key_file /home/ubuntu/fluentd/easy-rsa/easyrsa3/pki/private/client.key
      </tls>
    </security>
    <monitor>
      send_time true
    </monitor>  
    #<buffer>
      #@type file
      #path /var/log/td-agent/mqtt/topic/enrichportal/log
      #flush_interval 10s
    #</buffer>
  </match>
</label>
---

# start fluentd (input)
fluentd -v -c fluent-in_mqtt-tls.conf
# test publish to the topic “test”
mosquitto_pub -h localhost -p 8883 -t test -m "hoge" --cert easy-rsa/easyrsa3/pki/issued/client.crt --key easy-rsa/easyrsa3/pki/private/client.key --cafile easy-rsa/easyrsa3/pki/ca.crt

# start fluentd (output)
fluentd -v -c fluent-out_mqtt-tls.conf
# subscribe a topic test/rewritten by mosquitto_sub
mosquitto_sub -h localhost -p 8883 -t test/rewritten --cert easy-rsa/easyrsa3/pki/issued/client.crt --key easy-rsa/easyrsa3/pki/private/client.key --cafile easy-rsa/easyrsa3/pki/ca.crt
# test publish to the topic “test”
mosquitto_pub -h localhost -p 8883 -t test -m "hoge" --cert easy-rsa/easyrsa3/pki/issued/client.crt --key easy-rsa/easyrsa3/pki/private/client.key --cafile easy-rsa/easyrsa3/pki/ca.crt
toyokazu commented 4 years ago

If you want to test locally modified version from github image, please try "rake build" and "gem install pkg/fluent-plugin-mqtt-io-x.x.x.gem" to install the modified gem into your gem folder.

otokunaga2 commented 4 years ago

Hi, thank you for your kind and quick reply. I have totally resolved my issue with your advice. Thank you very much.

As you mentioned, the path issue is based on my environment, so I close both my PR and this issue.

Best,