Open quoccuongmmt opened 4 years ago
Check our FAQ. There are several approaches for it:
https://docs.fluentd.org/quickstart/faq#i-got-encoding-error-inside-the-plugin-how-to-fix-it
Check our FAQ. There are several approaches for it:
https://docs.fluentd.org/quickstart/faq#i-got-encoding-error-inside-the-plugin-how-to-fix-it
Hello. Sorry for the necro, but I figured it'd be better to posting in an already existing issue than creating a new one on the same topic.
We are facing a similar error as @quoccuongmmt:
#0 got unrecoverable error in primary and no secondary error_class=Encoding::UndefinedConversionError error="\"\\xC3\" from ASCII-8BIT to UTF-8"
I had a look at your link there @repeatedly and tried the record_modifier
solution, but sadly it didn't help (or quite possibly, I didn't use it correctly). We still get the same error after a while, and we have a backup directory filling up with bad chunks in files.
These files contain lines like this:
<92>Îf8g«<84>Ä^KdefaultData<86>Ä^Guser_idÎ^@^AU-Ä^Nsecurity_level<91>Ä^CpwdÄ^Mactive_idÀÄ^FactionÄ^QLogin: SuccessfulÄ
activeYearÍ^GèÄ^KactiveMonth^EÄ^Eextra<82>Ä^HusernameÄ^LMarOskuerrÄ^Kauth_methodÄ^HpasswordÄ^ElevelÄ^DINFO¤dist¤app<92>Îf8gî<84>Ä^KdefaultData<86>Ä^Guser_idÎ^@^Cµ|Ä^Nsecurity_level<91>Ä^CpwdÄ^Mactive_idÀÄ^FactionÄ^QLogin: SuccessfulÄ
This is our config:
<system>
log_level info
</system>
<source>
@type monitor_agent
bind 0.0.0.0
port 24220
</source>
<source>
@type forward
@id app
@label @app
port 24224
</source>
<label @app>
<match *.action.*>
@type mongo
host mongo_host
port 27017
database mongo_db
collection log_action
user mongo_user
password mongo_pwd
<buffer>
flush_interval 10s
</buffer>
<inject>
time_key time
</inject>
</match>
<match *.user.*>
@type mongo
host mongo_host
port 27017
database mongo_db
collection log_user
user mongo_user
password mongo_pwd
<buffer>
flush_interval 10s
</buffer>
<inject>
time_key time
</inject>
</match>
</label>
I tried two things with record_modifier
where I first put the following before the <label @app>
section:
...
<source>
@type forward
@id app
@label @app
port 24224
</source>
<filter **>
@type record_modifier
char_encoding utf-8
</filter>
<label @app>
<match *.action.*>
@type mongo
host mongo_host
...
When that didn't work, I tried the following instead:
...
<source>
@type forward
@id app
@label @app
port 24224
</source>
<filter **>
@type record_modifier
char_encoding ascii-8bit:utf-8
</filter>
<label @app>
<match *.action.*>
@type mongo
host mongo_host
...
But to no avail. Maybe I'm not understanding the instructions in the in the record_modifier
README or simply misunderstand how fluentd works (I am not all that familiar with the software). We would greatly appreciate any help/pointers for solving this issue.
OS: Ubuntu 22.04 td-agent 4.4.2 fluentd 1.15.3 gem fluent-plugin-mongo (1.6.0) gem mongo (2.18.2)
I wonder if there should be a .force_encoding "utf-8"
added somewhere in the code, similar to https://github.com/SpringMT/fluent-plugin-ses/pull/1/
My config td-agent
Some log code have special-character word like
{"character": "༂Sυɠαɾ ɗαɗү࿐"}
. This log can't push to mongo and I was checked file log td-agent.log:I'm using OS: Centos 7 td-agent 3 Can everyone help me?