search

fluent / fluent-plugin-kafka

Kafka input and output plugin for Fluentd
Other
303 stars 178 forks source link

source sequence is illegal/malformed utf-8, ignored JSON::GeneratorError, the log is lost #250

Closed eunsikpark closed 4 years ago

eunsikpark commented 5 years ago

Sometimes I get this message and data is lost

[error]: #0 source sequence is illegal/malformed utf-8, ignored JSON::GeneratorError. ignore record : 진 차원§i

<source> @type kafka_group brokers somehost:9092,somehost:9092,somehost:9092 consumer_group group topics topics add_prefix kafka </source>

LOCAL GEMS

activesupport (4.2.11.1) addressable (2.5.2) autoparse (0.3.3) aws-eventstream (1.0.1) aws-partitions (1.105.0) aws-sdk-core (3.30.0) aws-sdk-kms (1.9.0) aws-sdk-s3 (1.21.0) aws-sdk-sqs (1.7.0) aws-sigv4 (1.0.3) bigdecimal (default: 1.3.2) bundler (1.16.2) concurrent-ruby (1.1.5) cool.io (1.5.3) did_you_mean (1.1.0) dig_rb (1.0.1) digest-crc (0.4.1) elasticsearch (6.1.0) elasticsearch-api (6.1.0) elasticsearch-transport (6.1.0) excon (0.62.0) extlib (0.9.16) faraday (0.15.3) fluent-config-regexp-type (1.0.0) fluent-logger (0.7.2) fluent-mixin-config-placeholders (0.4.0) fluent-plugin-elasticsearch (2.11.11) fluent-plugin-forest (0.3.3) fluent-plugin-kafka (0.9.2) fluent-plugin-record-modifier (1.1.0) fluent-plugin-rewrite-tag-filter (2.1.0) fluent-plugin-s3 (1.1.6) fluent-plugin-td (1.0.0) fluent-plugin-td-monitoring (0.2.4) fluent-plugin-webhdfs (1.2.3) fluentd (1.2.6) google-api-client (0.8.7) googleauth (0.8.1) hirb (0.7.3) http_parser.rb (0.6.0) httpclient (2.8.2.4) i18n (0.9.5) io-console (default: 0.4.6) ipaddress (0.8.3) jmespath (1.4.0) json (default: 2.0.4) jwt (2.1.0) launchy (2.4.3) ltsv (0.1.0) memoist (0.16.0) mini_portile2 (2.3.0) minitest (5.10.1) mixlib-cli (1.7.0) mixlib-config (2.2.4) mixlib-log (1.7.1) mixlib-shellout (2.2.7) msgpack (1.2.4) multi_json (1.13.1) multipart-post (2.0.0) net-telnet (0.1.1) nokogiri (1.8.5) ohai (6.20.0) oj (3.3.9) openssl (default: 2.0.7) os (1.0.0) parallel (1.12.1) power_assert (0.4.1) psych (default: 2.2.2) public_suffix (3.0.3) rake (12.0.0) rdoc (default: 5.0.0) retriable (1.4.1) ruby-kafka (0.7.5) ruby-progressbar (1.10.0) rubyzip (1.2.2) serverengine (2.0.7) sigdump (0.2.4) signet (0.11.0) strptime (0.2.3) systemu (2.5.2) td (0.16.1) td-client (1.0.6) td-logger (0.3.27) test-unit (3.2.3) thread_safe (0.3.6) tzinfo (1.2.5) tzinfo-data (1.2018.5) uuidtools (2.1.5) webhdfs (0.8.0) xmlrpc (0.2.1) yajl-ruby (1.4.0) zip-zip (0.3) zookeeper (1.4.11)

repeatedly commented 5 years ago

Does this message happen in kafka plugin? I checked ruby-kafka and fluent-plugin-kafka but no ignore record in the implementation.