sampointer
commented
6 years ago If this is anything like fluentd, which uses msgpack, then small messages can blow ElasticSearch request limits if they pack very efficiently. In fluentd the message size limits are enforced after serialization to the transport format, which when unrolled at the other end can balloon to into huge HTTP POSTs, which are rejected. This is especially prevalent with JSON and logging information, which msgpack compresses very well.
The fluentd ecosystem is a tire fire, so don't be tempted to jump ship in that direction :smiley:
RT need any other config items ?
follow is my part config [hekad] maxprocs = 16 base_dir = "/export/home/hekad" max_message_size = 10485760
attack log default
[nginx_udp_551] type = "UdpInput" address = "172.18.182.162:551" decoder = "JsonDecoder" send_decode_failures = true log_decode_failures = true
[JsonDecoder] type = "SandboxDecoder" filename = "lua_decoders/json.lua"
[JsonDecoder.config] payload_keep = false map_fields = true Timestamp = "time_stamp" Type = "log_type"
type = "ngx_log"