Some messages can be dropped

[mixazya]

Issue happend when data coming compressed: Module used lib/fluent/plugin/socket_util.rb class UdpHandler which chomping incoming data:

      def on_readable
        msg, addr = @io.recvfrom_nonblock(@body_size_limit)
        msg.chomp!
        @callback.call(msg, addr)
      rescue => e
        @log.error "unexpected error", error: e, error_class: e.class
      end

So when you receiving compressed packet then ending with '0d' (CR), this symbol will be removed and compressed stream will be damaged. Testing with compressing of zlib, after symbol was removed, zlib failed to decode and show following message in logs: 2017-12-26 13:28:54 +1000 [warn]: Gelfd failed to parse a message error="Failed to decode data: buffer error "

One way of resolving:

--- /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-input-gelf-0.2.0/lib/fluent/plugin/in_gelf.rb.bkp 2017-12-26 13:16:37.532566124 +1000
+++ /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-input-gelf-0.2.0/lib/fluent/plugin/in_gelf.rb 2017-12-26 13:29:18.422364466 +1000
@@ -7,6 +7,21 @@
 require 'fluent/input'

 module Fluent
+#mixaz patch
+  class UdpHandler < SocketUtil::UdpHandler
+      def initialize(io, log, body_size_limit, callback)
+       super
+      end
+      def on_readable
+        msg, addr = @io.recvfrom_nonblock(@body_size_limit)
+        @callback.call(msg, addr)
+      rescue => e
+        @log.error "unexpected error", error: e, error_class: e.class
+      end
+  end
+#mixaz patch end
+
   class GelfInput < Fluent::Input
     Fluent::Plugin.register_input('gelf', self)

@@ -105,7 +120,10 @@
       else
         @usock = SocketUtil.create_udp_socket(@bind)
         @usock.bind(@bind, @port)
-        SocketUtil::UdpHandler.new(@usock, log, 8192, callback)
+#mixaz patch
+        #SocketUtil::UdpHandler.new(@usock, log, 8192, callback)
+        UdpHandler.new(@usock, log, 8192, callback)
+#mixaz patch end
       end
     end

[wdoekes]

This bug fix is not necessary anymore. It was fixed in https://github.com/MerlinDMC/fluent-plugin-input-gelf/commit/fb0b653c051e8696ae988daa43f607c51356f03d when the Fluent::SocketUtil::UdpHandler was replaced by server_create. The UdpHandler did chomping, while the server_create callback handler does not:

https://github.com/fluent/fluentd/blob/5844f7209fec154a4e6807eb1bee6989d3f3297f/lib/fluent/compat/socket_util.rb#L36-L52

vs.

https://github.com/fluent/fluentd/blob/b04e8084a887dc8395d49c606f474e33a396997c/lib/fluent/plugin_helper/server.rb#L178 https://github.com/fluent/fluentd/blob/5844f7209fec154a4e6807eb1bee6989d3f3297f/lib/fluent/plugin/in_udp.rb#L72-L74

[wdoekes]

(A bug fix that is necessary, is #16 though. Also causing messages to get dropped.)