Fluentd 14.0. v SSLErrorWaitReadable error with SecureForward plugin

[vishnuvgdemos]

I tried using Fluentd v 14.0.1 with

ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux](Installed using RhRuby 2.2) OpenSSL 1.0.1e-fips 11 Feb 2013

I am getting error [Fluent::SecureForwardOutput] SSLError error_class=OpenSSL::SSL::SSLErrorWaitReadable error=# mtime=2016-07-25 08:34:52 +0000 host="10.4.1.10" port=24284

Fluentd using fluent-plugin-secure-forward' version '0.4.2 with the correct certificates to transfer the data from forwarder to aggregator. But the above issue is happening even after updating the openssl versions to latest one.

[repeatedly]

Also refer to https://groups.google.com/forum/#!topic/fluentd/amHr3W2qMKc out_secure_forward should catch OpenSSL::SSL::SSLErrorWaitReadable and retry.

[tagomoris]

Can you see where that error was raised from? As far as I checked, that error should be rescued by these code: https://github.com/tagomoris/fluent-plugin-secure-forward/blob/master/lib/fluent/plugin/output_node.rb#L294 https://github.com/tagomoris/fluent-plugin-secure-forward/blob/master/lib/fluent/plugin/output_node.rb#L314 https://github.com/tagomoris/fluent-plugin-secure-forward/blob/master/lib/fluent/plugin/output_node.rb#L345

[vishnuvgdemos]

Hi Tagomoris,

The warning is happening at forwarder level.

The warning which I mentioned [Fluent::SecureForwardOutput] SSLError error_class=OpenSSL::SSL::SSLErrorWaitReadable error=# mtime=2016-07-25 08:34:52 +0000 host="10.4.1.10" port=24284 is getting at forwarders.

At the aggregator I can see from the trace that the forwarder was able to connect to aggregator , but no data transfer happening through secure forward plugin. And on the forwarder its showing warning " [Fluent::SecureForwardOutput] SSLError error_class=OpenSSL::SSL::SSLErrorWaitReadable error=# mtime=2016-07-25 08:34:52 +0000 host="10.4.1.10" port=24284"

More Details

The version is 'fluent-plugin- version secure-forward' '0.4.2'

[tagomoris]

@vishnuvg1984 Thank you for responding my question. What I need is full backtrace (if exists), or raw log whole line which includes the message you pasted. Could you paste it?

[vishnuvgdemos]

Hi Tagomoris,

@tagomoris Thanks, Sure I can attach both the forwarder and aggregator tracelog . Please find the attached files AggregatorLog.txt ForwarderLog.txt

Also Please find the environment version details.

Forwarder & Aggregator OS : CentOS Linux release 7.2.1511 (Core) FLuentdv 14.0.1 Open SSL : OpenSSL 1.0.1e-fips 11 Feb 2013

Please let me know if require any other details.

[tagomoris]

Oh, are you mentioning about trace logs? It's very normal and are handled by out_secure_forward code. There are no problem. Trace logs are printed to know where or when problems occurs, and that SSLError you mentioned doesn't cause any problems.

I think the problem you have is completely different one (like: any events are not transferred from forwarder to aggregator even when connection established). Please re-check your tag matching in forwarder configuration file, or insert <match *.**> @type stdout</match> to show events in your forwarder processes.

This case is not a bug, nor any problems.

[briandeheus]

I'm having the same issue as @vishnuvg1984. I've added a match to stdout and can confirm the messages are showing up in the log files of the forwarder.

Other than that there is a whole lot of:

2016-08-23 23:55:18 -0400 fluent.trace: {"error_class":"OpenSSL::SSL::SSLErrorWaitReadable","error":"#<OpenSSL::SSL::SSLErrorWaitReadable: read would block>","mtime":"2016-08-23 23:49:59 -0400","host":"xxx.xxx.jp","port":24284,"message":"SSLError error_class=OpenSSL::SSL::SSLErrorWaitReadable error=#<OpenSSL::SSL::SSLErrorWaitReadable: read would block> mtime=2016-08-23 23:49:59 -0400 host=\"xxx.xxx.jp\" port=24284"}

using a similar setup as @vishnuvg1984

[emayssat-ms]

Same issue here, when log_level is set to trace. Although that may not be an issue, the log file is then meaningless since this message is hiding other meaningful ones.

[jonapich]

Same discovery on my end; tried enabling -vv to get some trace for another problem and the output is completely flooded with these messages.

@tagomoris any chance we can get fewer messages or something? :(