search

tagomoris / fluent-plugin-secure-forward

Other
140 stars 30 forks source link

no one nodes with valid ssl session #15

Closed smcavoy-b closed 8 years ago

smcavoy-b commented 9 years ago

Hello, I have configured 3 sessions per system (there are 13 systems). I am randomly getting the error in the subject. The error occurs on one of the sessions (configured match) while the other two are perfectly fine. Restarting does not work, I must kill -9 to get it to stop. Once started the buffers on disk are loaded and sent. I have included the log entries below. I am running 0.2.6 and using td-agent-2.2.0 which uses ruby 2.1.5p273

Any suggestions?

2015-04-07 14:35:12 +0000 [trace]: plugin/out_secure_forward.rb:139:block in node_watcher: in node health watcher 2015-04-07 14:35:12 +0000 [trace]: plugin/out_secure_forward.rb:142:block (2 levels) in node_watcher: node health watcher for lumberjack.advidi.com 2015-04-07 14:35:13 +0000 [trace]: plugin/out_secure_forward.rb:218:write_objects: selected node host="lumberjack.advidi.com" port=24284 standby=false 2015-04-07 14:35:15 +0000 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2015-04-07 14:51:37 +0000 error_class="RuntimeError" error="no one nodes with valid ssl session" plugin_id="campaign_secure_tracking_clicks" 2015-04-07 14:35:15 +0000 [warn]: suppressed same stacktrace

tagomoris commented 9 years ago

Hmm, look very curious. @smcavoy-b Do you have systems running fluent-plugin-secure-forward with td-agent 2.1? Doesn't any problems happen on that nodes?

repeatedly commented 9 years ago

I want to know this issue happens on only td-agent 2.2.0 or not. It helps investingating this problem.

smcavoy-b commented 9 years ago

I have reverted one of our systems to td-agent 2.1 and run some basic tests and found no error. Unfortunately the problem does not seem to be easily reproducible.

On 7 April 2015 at 23:38, Masahiro Nakagawa notifications@github.com wrote:

I want to know this issue happens on only td-agent 2.2.0 or not. It helps investingating this problem.

— Reply to this email directly or view it on GitHub https://github.com/tagomoris/fluent-plugin-secure-forward/issues/15#issuecomment-90739814 .

repeatedly commented 9 years ago

I see. What the distribution do you use? CentOS, Ubuntu or Debian?

@tagomoris Can we know the cause of error by applying patch? For example, dumping an exceptions or check the OpenSSL errors.

tagomoris commented 9 years ago

I pushed a branch w/ some additional logging. This code may help investigation... https://github.com/tagomoris/fluent-plugin-secure-forward/tree/paranoid-logging-output-session

tagomoris commented 8 years ago

I'll close this issue at once. Please reopen this and notify me with DEBUG/TRACE logs if you have further problems.