Getting too many connection resets for fluentd-plugin-splunk-hec

[shrinaththube]

I am using splunk/splunk docker image. I have generated hec token and kept SSL enable. I have generated self-signed certs similar as following documentation https://docs.splunk.com/Documentation/Splunk/7.2.5/Security/Howtoself-signcertificates https://docs.splunk.com/Documentation/Splunk/7.2.5/Security/HowtoprepareyoursignedcertificatesforSplunk https://docs.splunk.com/Documentation/Splunk/7.2.5/Security/ConfigureSplunkforwardingtousesignedcertificates

Modified input.conf

[splunktcp-ssl:8088]
disabled=0
[SSL]
serverCert = $SPLUNK_HOME/etc/auth/myNewServerCertificate.pem
sslPassword = <password>
rootCA= $SPLUNK_HOME/etc/auth/myCACertificate.pem
requireClientCert = false

Fluentd configuration

  <match **>
    @type splunk_hec
    hec_host <VM_Hostname_Where_Splunk_Container_is_Running>
    hec_port 8088
    hec_token <ssl_token>
    insecure_ssl true
    idle_timeout 10
    read_timeout 5
    ca_file "/fluentd/etc/tls/cacert.pem"
    <buffer>
      flush_thread_count 1
      flush_interval 65s
      chunk_limit_size 1M
      queue_limit_length 32
      retry_max_interval 30
      retry_forever false
    </buffer>
  </match>

I have kept server.pem certs CN as VM hostname where splunk container is running

After all these steps, I am getting following error at Fluentd side -

2019-03-21 23:43:08 +0000 [warn]: #0 failed to flush the buffer. retry_time=21 next_retry_seconds=2019-03-23 00:00:19 +0000 chunk="584b78a80f58364374f77ae06e4a0ebc" error_class=Net::HTTP::Persistent::Error error="too many connection resets (due to Connection reset by peer - Errno::ECONNRESET) after 0 requests on 69851140794400, last used 27.151896316 seconds ago"

And getting following error in a spunkd.log -

03-21-2019 23:44:08.343 +0000 DEBUG TcpInputConfig - connection_host=ip for <IP_Address>
03-21-2019 23:44:08.345 +0000 ERROR TcpInputProc - Message rejected. Received unexpected message of size=1347375956 bytes from src=<IP_Address> in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.
03-21-2019 23:44:41.009 +0000 DEBUG TcpInputConfig - connection_host=ip for <IP_Address>

I am sending only a single line log which is way below the data limits. I am getting this problem only for HTTPS and not for HTTP. I am also able to send testing logs using CURL.

Can you please let me know if I am misconfiguring something?

[dtregonning]

Hi @shrinaththube from this Error its looks likes you are trying to send too many things at once. Id look to reduce the chunk_limit_size 1M in your fluent configuration and try again

[shrinaththube]

@dtregonning Thank you for responding. I am actually only sending a single line log as { "key" : "Hello" } and in a fluentd configuration I have kept chunk_limit_size 1M

[dtregonning]

@shrinaththube Understood. Which protocol do you have set on your Splunk HEC token? See attached image (Enable SSL Option)

You may also look to add the protocol parameter in your Fluentd configuration. By default this is HTTPS. You'll need to ensure this matches.

Are you seeing any data flow through at all? Apart from test data?

[dtregonning]

Also noting your inputs.conf configuration supplied. Using HEC will be set up in a different stanza not [splunktcp-ssl:8088] - https://docs.splunk.com/Documentation/Splunk/7.2.5/Admin/Inputsconf - see HTTP Event Collector Section.

[shrinaththube]

@dtregonning Thank you for pointing in the right direction. I update the input.conf file and it worked. Here is the configuration I have used -

[http]
port = 8088
disabled = 0
enableSSL = 1
dedicatedIoThreads = 4
maxSockets = 50
maxThreads = 20
serverCert = /opt/splunk/etc/auth/myNewServerCertificate.pem
sslPassword = <password>
caCertFile = /opt/splunk/etc/auth/myCACertificate.pem

Thank you once again!! Closing the issue.