fluent / fluent-plugin-splunk

Fluentd Plugin for Splunk
Apache License 2.0
41 stars 19 forks source link

Configurable metadata fields for TCP output plugin #33

Open cotej opened 4 years ago

cotej commented 4 years ago

This is a feature request to allow configurable default metadata fields (host, source, sourcetype, index) for the TCP output, as is currently supported by the HEC output.

I believe this should be possible by sending a header over the TCP socket as the first line of the payload, which should look like:

***SPLUNK*** <metadata field>=<string> <metadata field>=<string> ...

More info at: https://docs.splunk.com/Documentation/Splunk/8.0.4/Data/Assignmetadatatoeventsdynamically

cotej commented 4 years ago

I'm thinking something along the lines of this oughta do it. Seems to be working as expected from some basic testing.

--- a/lib/fluent/plugin/out_splunk_tcp.rb
+++ b/lib/fluent/plugin/out_splunk_tcp.rb
@@ -14,6 +14,11 @@ module Fluent
     config_param :host, :string
     config_param :port, :integer

+    config_param :default_host, :string, default: nil
+    config_param :default_source, :string, default: nil
+    config_param :default_sourcetype, :string, default: nil
+    config_param :default_index, :string, default: nil
+
     config_param :format, :string, default: 'raw'

     # for raw format
@@ -74,6 +79,20 @@ module Fluent
       else
         raise ConfigError, "invalid 'format' option: #{@format}"
       end
+
+      @headers = []
+      if @default_host
+        @headers << "host=#{@default_host}"
+      end
+      if @default_source
+        @headers << "source=#{@default_source}"
+      end
+      if @default_sourcetype
+        @headers << "sourcetype=#{@default_sourcetype}"
+      end
+      if @default_index
+        @headers << "index=#{@default_index}"
+      end
     end

     def multi_workers_ready?
@@ -92,6 +111,11 @@ module Fluent
       return if chunk.empty?

       payload = ''
+      unless @headers.empty?
+        payload << "***SPLUNK*** #{@headers.join(' ')}"
+        payload << @line_breaker
+      end
+
       chunk.msgpack_each do |time, record|
         event = @formatter.call(time, record)
         unless event.empty?