search

fluent / fluent-plugin-mongo

MongoDB input and output plugin for Fluentd
https://docs.fluentd.org/output/mongo
173 stars 61 forks source link

Linux fluend mogon "\xE7" to UTF-8 in conversion from ASCII-8BIT,Mongodb cannot be written in Chinese #169

Open qaz63700 opened 2 years ago

qaz63700 commented 2 years ago

Describe the bug

Linux fluend mogon "\xE7" to UTF-8 in conversion from ASCII-8BIT,Mongodb cannot be written in Chinese

To Reproduce

Mongodb cannot be written in Chinese

Expected behavior

Chinese string successfully written to mongodb

Your Environment

- Fluentd version:1.4.6
- TD Agent version:center07
- Operating system:
- Kernel version:

Your Configuration

<source>
  @type tail
  path /home/admin/logs/manage.json.log.%Y-%m-%d
  <parse>
    @type json
  </parse>
  tag mongo.json.access
</source>
<filter >
  @type parser
  key_name message
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
  </parse>
</filter>

# <match >
#     @type stdout
# </match>
<match mongo.**>
  # plugin type
  @type mongo

  # mongodb db + collection
  database apache
  collection points

  # mongodb host + port
  host localhost
  port 27017

  # interval
  <buffer>
    flush_interval 30s
  </buffer>

  # # make sure to include the time key
  <inject>
    time_key time
  </inject>
</match>

Your Error Log

2022-05-19 14:06:39 +0800 [warn]: #0 got unrecoverable error in primary and no secondary error_class=Encoding::UndefinedConversionError error="\"\\xE7\" from ASCII-8BIT to UTF-8"
2022-05-19 14:06:39 +0800 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'got unrecoverable error in primary and no secondary error_class=Encoding::UndefinedConversionError error=\"\\\"\\\\xE7\\\" from ASCII-8BIT to UTF-8\"'" location=nil tag="fluent.warn" time=2022-05-19 14:06:39.439858832 +0800 record={"error"=>"#<Encoding::UndefinedConversionError: \"\\xE7\" from ASCII-8BIT to UTF-8>", "message"=>"got unrecoverable error in primary and no secondary error_class=Encoding::UndefinedConversionError error=\"\\\"\\\\xE7\\\" from ASCII-8BIT to UTF-8\""}
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/bson-4.15.0/lib/bson/string.rb:50:in `encode'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/bson-4.15.0/lib/bson/string.rb:50:in `put_string'
2022-05-19 14:06:39 +0800 [warn]: #0 no patterns matched tag="fluent.warn"
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/bson-4.15.0/lib/bson/string.rb:50:in `to_bson'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/bson-4.15.0/lib/bson/hash.rb:43:in `put_hash'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/bson-4.15.0/lib/bson/hash.rb:43:in `to_bson'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:326:in `serialize'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:287:in `block in serialize'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:286:in `each'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:286:in `serialize'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:175:in `block in serialize'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:168:in `each'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/serializers.rb:168:in `serialize'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/message.rb:241:in `block in serialize_fields'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/message.rb:229:in `each'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/message.rb:229:in `serialize_fields'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/message.rb:125:in `serialize'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/protocol/msg.rb:103:in `serialize'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection.rb:275:in `block in write'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection.rb:274:in `each'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection.rb:274:in `write'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection.rb:229:in `deliver'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection.rb:144:in `block in dispatch'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/monitoring/publishable.rb:48:in `publish_command'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection.rb:143:in `dispatch'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/operation/shared/executable.rb:34:in `block in dispatch_message'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server/connection_pool.rb:110:in `with_connection'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/server.rb:251:in `with_connection'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/operation/shared/executable.rb:33:in `dispatch_message'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/operation/insert/op_msg.rb:42:in `execute'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/operation/shared/write.rb:59:in `bulk_execute'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:210:in `insert_one'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:178:in `execute_operation'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:62:in `block (3 levels) in execute'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/retryable.rb:158:in `legacy_write_with_retry'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/retryable.rb:106:in `write_with_retry'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:61:in `block (2 levels) in execute'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:59:in `each'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:59:in `block in execute'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/cluster.rb:532:in `with_session'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/client.rb:550:in `with_session'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/bulk_write.rb:58:in `execute'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/collection.rb:532:in `bulk_write'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/mongo-2.6.4/lib/mongo/collection.rb:509:in `insert_many'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/fluent-plugin-mongo-1.5.0/lib/fluent/plugin/out_mongo.rb:334:in `operate'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/fluent-plugin-mongo-1.5.0/lib/fluent/plugin/out_mongo.rb:197:in `write'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'
  2022-05-19 14:06:39 +0800 [warn]: #0 /usr/local/rvm/gems/ruby-2.6.1/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-05-19 14:06:39 +0800 [warn]: #0 bad chunk is moved to /tmp/fluent/backup/worker0/object_14a7738/5df572b61d8caab81b9cad3613c6bb26.log
2022-05-19 14:06:39 +0800 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data 'bad chunk is moved to /tmp/fluent/backup/worker0/object_14a7738/5df572b61d8caab81b9cad3613c6bb26.log'" location=nil tag="fluent.warn" time=2022-05-19 14:06:39.440568521 +0800 record={"message"=>"bad chunk is moved to /tmp/fluent/backup/worker0/object_14a7738/5df572b61d8caab81b9cad3613c6bb26.log"}

Additional context

No response