Client raises Errno::ECONNRESET after tailing logs for 1 hour

[eljeffeg]

While monitoring "heroku logs --tail", I keep getting this issue.

! Heroku client internal error. ! Search for help at: https://help.heroku.com ! Or report a bug at: https://github.com/heroku/heroku/issues/new

Error:       Connection reset by peer (Errno::ECONNRESET)
Backtrace:   /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:104:in `read_all'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2228:in `read_body_0'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2181:in `read_body'
             /Users/Jeff/.heroku/client/lib/heroku/client.rb:524:in `read_logs'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1054:in `__request__'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2144:in `reading_body'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1053:in `__request__'
             /Users/Jeff/.heroku/client/vendor/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in `request'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:948:in `request_get'
             /Users/Jeff/.heroku/client/lib/heroku/client.rb:523:in `read_logs'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:543:in `start'
             /Users/Jeff/.heroku/client/lib/heroku/client.rb:522:in `read_logs'
             /Users/Jeff/.heroku/client/lib/heroku/command/logs.rb:35:in `index'
             /Users/Jeff/.heroku/client/lib/heroku/command.rb:206:in `send'
             /Users/Jeff/.heroku/client/lib/heroku/command.rb:206:in `run'
             /Users/Jeff/.heroku/client/lib/heroku/cli.rb:28:in `start'
             /usr/bin/heroku:28

Command:     heroku logs --app historylink --tail
Version:     heroku-gem/2.33.0 (universal-darwin12.0) ruby/1.8.7

[wuputah]

Hi- are you still having this issue?

[eljeffeg]

I don't usually leave it sitting, so it's not something I see often.. I run the logs, watch it for a few minutes, and then disconnect. I just fired it up though and let it sit for an hour and got the same error.

I was running heroku 2.33.1 on OS X 10.8.2. I'm updating to 2.33.3 and going to run it again and see.. will report back.

Error:       Connection reset by peer (Errno::ECONNRESET)
Backtrace:   /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:135:in `sysread'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:135:in `rbuf_fill'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:104:in `read_all'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2228:in `read_body_0'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2181:in `read_body'
             /Users/Jeff/.heroku/client/lib/heroku/client.rb:524:in `read_logs'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1054:in `__request__'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2144:in `reading_body'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1053:in `__request__'
             /Users/Jeff/.heroku/client/vendor/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in `request'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:948:in `request_get'
             /Users/Jeff/.heroku/client/lib/heroku/client.rb:523:in `read_logs'
             /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:543:in `start'
             /Users/Jeff/.heroku/client/lib/heroku/client.rb:522:in `read_logs'
             /Users/Jeff/.heroku/client/lib/heroku/command/logs.rb:35:in `index'
             /Users/Jeff/.heroku/client/lib/heroku/command.rb:206:in `send'
             /Users/Jeff/.heroku/client/lib/heroku/command.rb:206:in `run'
             /Users/Jeff/.heroku/client/lib/heroku/cli.rb:28:in `start'
             /usr/bin/heroku:28

Command:     heroku logs --tail
Version:     heroku-gem/2.33.1 (universal-darwin12.0) ruby/1.8.7

[wuputah]

Oh, you're leaving it running for an hour? We're probably disconnecting you in that case, hence the error, but that's not handled by client. @archaelus, can you comment on the behavior here?

[eljeffeg]

Ah, understandable. If that's the case, perhaps you can send a signal to the client so that it gracefully terminates with a user friendly message.

[archaelus]

Yup - that's a pretty terrible error message.

This is a consequence of our load balancers in front of Logplex instances - they sever the connection after an hour. We should change the client ideally to reconnect, but fixing the return message too is good.

[wuputah]

Thanks Geoff; glad to hear this is pretty much expected behavior. I think we just need to rescue this error appropriately in the client. Tailing logs for an hour seems like a very reasonable timeout to me.