search

sparklemotion / mechanize

Mechanize is a ruby library that makes automated web interaction easy.
https://www.rubydoc.info/gems/mechanize/
MIT License
4.4k stars 472 forks source link

too many connection resets (due to Timeout::Error - Timeout::Error) after 0 requests on ... #228

Closed mamantoha closed 12 years ago

mamantoha commented 12 years ago

After upgrading from 2.4 to 2.5 I am getting the error when log in to site.

/home/mama/.rvm/gems/ruby-1.9.3-p194/gems/net-http-persistent-2.6/lib/net/http/persistent.rb:839:in `rescue in request': too many connection resets (due to Timeout::Error - Timeout::Error) after 0 requests on 70037670, last used 1336776917.2502944 seconds ago (Net::HTTP::Persistent::Error)

Here is my code:

# -*- encoding: utf-8 -*-

require 'bundler'
Bundler.setup :default

require 'mechanize'
require 'logger'

puts Mechanize::VERSION

#log = Logger.new($stderr)
#log.level = Logger::DEBUG

agent = Mechanize.new do |a|
  #a.log = log
end

agent.agent.http.debug_output = $stderr

page = agent.get('http://vk.com/')
login_form = page.form_with(name: 'login')
login_form.email = 'anton.maminov@gmail.com'
login_form.pass = 'pass4mechanize'
page = login_form.submit

Debug log:

opening connection to vk.com...
opened
<- "GET / HTTP/1.1\r\nAccept: */*\r\nUser-Agent: Mechanize/2.5 Ruby/1.9.3p194 (http://github.com/tenderlove/mechanize/)\r\nAccept-Encoding: gzip,deflate,identity\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Language: en-us,en;q=0.5\r\nHost: vk.com\r\nConnection: keep-alive\r\nKeep-Alive: 300\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "Server: nginx/1.0.11\r\n"
-> "Date: Fri, 11 May 2012 22:53:16 GMT\r\n"
-> "Content-Type: text/html; charset=windows-1251\r\n"
-> "Connection: keep-alive\r\n"
-> "X-Powered-By: PHP/5.3.3-7+squeeze7\r\n"
-> "Set-Cookie: remixlang=3; expires=Sun, 19-May-2013 02:00:30 GMT; path=/; domain=.vk.com\r\n"
-> "Set-Cookie: remixchk=5; expires=Tue, 21-May-2013 01:50:07 GMT; path=/; domain=.vk.com\r\n"
-> "Pragma: no-cache\r\n"
-> "Cache-control: no-store\r\n"
-> "Content-Encoding: gzip\r\n"
-> "Vary: Accept-Encoding\r\n"
-> "Content-Length: 4895\r\n"
-> "\r\n"
reading 4895 bytes...
[...]
read 4895 bytes
Conn keep-alive
opening connection to login.vk.com...
opened
<- "POST /?act=login HTTP/1.1\r\nAccept: */*\r\nUser-Agent: Mechanize/2.5 Ruby/1.9.3p194 (http://github.com/tenderlove/mechanize/)\r\nAccept-Encoding: gzip,deflate,identity\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Language: en-us,en;q=0.5\r\nCookie: remixlang=3; remixchk=5\r\nHost: login.vk.com\r\nReferer: http://vk.com/\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: 170\r\nConnection: keep-alive\r\nKeep-Alive: 300\r\n\r\n"
<- "act=login&q=1&al_frame=1&expire=&captcha_sid=&captcha_key=&from_host=vk.com&from_protocol=http&ip_h=0af0ad5b6345fa61d1&email=anton.maminov%40gmail.com&pass=pass4mechanize"
-> "HTTP/1.1 302 Found\r\n"
-> "Server: nginx/1.0.11\r\n"
-> "Date: Fri, 11 May 2012 22:53:16 GMT\r\n"
-> "Content-Type: text/html; charset=windows-1251\r\n"
-> "Connection: keep-alive\r\n"
-> "X-Powered-By: PHP/5.2.6-1+lenny8\r\n"
-> "Pragma: no-cache\r\n"
-> "Cache-control: no-store\r\n"
-> "P3P: CP=\"IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT\"\r\n"
-> "Set-Cookie: s=1; expires=Fri, 10-May-2013 09:10:54 GMT; path=/; domain=login.vk.com\r\n"
-> "Set-Cookie: l=83380724; expires=Sun, 05-May-2013 12:16:35 GMT; path=/; domain=login.vk.com\r\n"
-> "Set-Cookie: p=41e4d29bdbc9a0b856063a4b2708a7a0617d; expires=Thu, 02-May-2013 08:01:15 GMT; path=/; domain=login.vk.com\r\n"
-> "Location: http://vk.com/login.php?act=slogin&al_frame=1&hash=36e3c2d61e19ba395f3369d0da7d6d48&s=1\r\n"
-> "Content-Encoding: gzip\r\n"
-> "Vary: Accept-Encoding\r\n"
-> "Content-Length: 26\r\n"
-> "\r\n"
reading 26 bytes...
-> "\x1F\x8B\b\x00\x00\x00\x00\x00\x00\x03\x02\x00\x00\x00\xFF\xFF\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00"
read 26 bytes
Conn keep-alive
<- "GET /login.php?act=slogin&al_frame=1&hash=36e3c2d61e19ba395f3369d0da7d6d48&s=1 HTTP/1.1\r\nAccept: */*\r\nUser-Agent: Mechanize/2.5 Ruby/1.9.3p194 (http://github.com/tenderlove/mechanize/)\r\nAccept-Encoding: gzip,deflate,identity\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Language: en-us,en;q=0.5\r\nCookie: remixlang=3; remixchk=5\r\nHost: vk.com\r\nReferer: http://vk.com/\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: 170\r\nConnection: keep-alive\r\nKeep-Alive: 300\r\n\r\n"
Conn close because of error end of file reached
opening connection to vk.com...
opened
<- "GET /login.php?act=slogin&al_frame=1&hash=36e3c2d61e19ba395f3369d0da7d6d48&s=1 HTTP/1.1\r\nAccept: */*\r\nUser-Agent: Mechanize/2.5 Ruby/1.9.3p194 (http://github.com/tenderlove/mechanize/)\r\nAccept-Encoding: gzip,deflate,identity\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Language: en-us,en;q=0.5\r\nCookie: remixlang=3; remixchk=5\r\nHost: vk.com\r\nReferer: http://vk.com/\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: 170\r\nConnection: keep-alive\r\nKeep-Alive: 300\r\n\r\n"
Conn close because of error Timeout::Error
/home/mama/.rvm/gems/ruby-1.9.3-p194/gems/net-http-persistent-2.6/lib/net/http/persistent.rb:839:in `rescue in request': too many connection resets (due to Timeout::Error - Timeout::Error) after 0 requests on 70037670, last used 1336776917.2502944 seconds ago (Net::HTTP::Persistent::Error)
        from /home/mama/.rvm/gems/ruby-1.9.3-p194/gems/net-http-persistent-2.6/lib/net/http/persistent.rb:848:in `request'
        from /home/mama/.rvm/gems/ruby-1.9.3-p194/gems/mechanize-2.5/lib/mechanize/http/agent.rb:258:in `fetch'
        from /home/mama/.rvm/gems/ruby-1.9.3-p194/gems/mechanize-2.5/lib/mechanize/http/agent.rb:944:in `response_redirect'
        from /home/mama/.rvm/gems/ruby-1.9.3-p194/gems/mechanize-2.5/lib/mechanize/http/agent.rb:299:in `fetch'
        from /home/mama/.rvm/gems/ruby-1.9.3-p194/gems/mechanize-2.5/lib/mechanize.rb:1229:in `post_form'
        from /home/mama/.rvm/gems/ruby-1.9.3-p194/gems/mechanize-2.5/lib/mechanize.rb:515:in `submit'
        from /home/mama/.rvm/gems/ruby-1.9.3-p194/gems/mechanize-2.5/lib/mechanize/form.rb:178:in `submit'
        from vk.rb:24:in `<main>'

With Mechanize 2.4

opening connection to vk.com...
opened
<- "GET / HTTP/1.1\r\nAccept: */*\r\nUser-Agent: Mechanize/2.4 Ruby/1.9.3p194 (http://github.com/tenderlove/mechanize/)\r\nAccept-Encoding: gzip,deflate,identity\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Language: en-us,en;q=0.5\r\nHost: vk.com\r\nConnection: keep-alive\r\nKeep-Alive: 300\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "Server: nginx/1.0.11\r\n"
-> "Date: Fri, 11 May 2012 23:02:22 GMT\r\n"
-> "Content-Type: text/html; charset=windows-1251\r\n"
-> "Connection: keep-alive\r\n"
-> "X-Powered-By: PHP/5.3.3-7+squeeze8\r\n"
-> "Set-Cookie: remixlang=3; expires=Sat, 04-May-2013 06:17:43 GMT; path=/; domain=.vk.com\r\n"
-> "Set-Cookie: remixchk=5; expires=Tue, 07-May-2013 15:17:49 GMT; path=/; domain=.vk.com\r\n"
-> "Pragma: no-cache\r\n"
-> "Cache-control: no-store\r\n"
-> "Content-Encoding: gzip\r\n"
-> "Vary: Accept-Encoding\r\n"
-> "Content-Length: 4896\r\n"
-> "\r\n"
reading 4896 bytes...
[...]
read 4896 bytes
Conn keep-alive
opening connection to login.vk.com...
opened
<- "POST /?act=login HTTP/1.1\r\nAccept: */*\r\nUser-Agent: Mechanize/2.4 Ruby/1.9.3p194 (http://github.com/tenderlove/mechanize/)\r\nAccept-Encoding: gzip,deflate,identity\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Language: en-us,en;q=0.5\r\nCookie: remixlang=3; remixchk=5\r\nHost: login.vk.com\r\nReferer: http://vk.com/\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: 170\r\nConnection: keep-alive\r\nKeep-Alive: 300\r\n\r\n"
<- "act=login&q=1&al_frame=1&expire=&captcha_sid=&captcha_key=&from_host=vk.com&from_protocol=http&ip_h=0af0ad5b6345fa61d1&email=anton.maminov%40gmail.com&pass=pass4mechanize"
-> "HTTP/1.1 302 Found\r\n"
-> "Server: nginx/1.0.11\r\n"
-> "Date: Fri, 11 May 2012 23:02:23 GMT\r\n"
-> "Content-Type: text/html; charset=windows-1251\r\n"
-> "Connection: keep-alive\r\n"
-> "X-Powered-By: PHP/5.2.6-1+lenny9\r\n"
-> "Pragma: no-cache\r\n"
-> "Cache-control: no-store\r\n"
-> "P3P: CP=\"IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT\"\r\n"
-> "Set-Cookie: s=1; expires=Sun, 05-May-2013 17:57:08 GMT; path=/; domain=login.vk.com\r\n"
-> "Set-Cookie: l=83380724; expires=Mon, 06-May-2013 02:46:39 GMT; path=/; domain=login.vk.com\r\n"
-> "Set-Cookie: p=fa9010a03738a9f1bede70546af85b51a19a; expires=Mon, 20-May-2013 16:17:08 GMT; path=/; domain=login.vk.com\r\n"
-> "Location: http://vk.com/login.php?act=slogin&al_frame=1&hash=2b89de7e7acf901c0be704d58a5e9545&s=1\r\n"
-> "Content-Encoding: gzip\r\n"
-> "Vary: Accept-Encoding\r\n"
-> "Content-Length: 26\r\n"
-> "\r\n"
reading 26 bytes...
-> "\x1F\x8B\b\x00\x00\x00\x00\x00\x00\x03\x02\x00\x00\x00\xFF\xFF\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00"
read 26 bytes
Conn keep-alive
<- "GET /login.php?act=slogin&al_frame=1&hash=2b89de7e7acf901c0be704d58a5e9545&s=1 HTTP/1.1\r\nAccept: */*\r\nUser-Agent: Mechanize/2.4 Ruby/1.9.3p194 (http://github.com/tenderlove/mechanize/)\r\nAccept-Encoding: gzip,deflate,identity\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Language: en-us,en;q=0.5\r\nCookie: remixlang=3; remixchk=5\r\nHost: vk.com\r\nReferer: http://vk.com/\r\nConnection: keep-alive\r\nKeep-Alive: 300\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "Server: nginx/1.0.11\r\n"
-> "Date: Fri, 11 May 2012 23:02:23 GMT\r\n"
-> "Content-Type: text/html; charset=windows-1251\r\n"
-> "Connection: keep-alive\r\n"
-> "X-Powered-By: PHP/5.2.6-1+lenny9\r\n"
-> "Pragma: no-cache\r\n"
-> "Cache-control: no-store\r\n"
-> "Set-Cookie: remixlang=1; expires=Mon, 06-May-2013 10:44:47 GMT; path=/; domain=.vk.com\r\n"
-> "P3P: CP=\"IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT\"\r\n"
-> "Set-Cookie: remixreg_sid=deleted; expires=Thu, 12-May-2011 23:02:22 GMT; path=/; domain=.vk.com\r\n"
-> "Set-Cookie: remixrec_sid=deleted; expires=Thu, 12-May-2011 23:02:22 GMT; path=/; domain=.vk.com\r\n"
-> "Set-Cookie: remixapi_sid=deleted; expires=Thu, 12-May-2011 23:02:22 GMT; path=/; domain=.vk.com\r\n"
-> "Set-Cookie: remixsid=b4e1ad2e88d67fbf6414a279c7df599336b6694412697547c2e3927eab85; expires=Thu, 16-May-2013 08:05:53 GMT; path=/; domain=.vk.com\r\n"
-> "Content-Encoding: gzip\r\n"
-> "Vary: Accept-Encoding\r\n"
-> "Content-Length: 446\r\n"
-> "\r\n"
reading 446 bytes...
[...]
read 446 bytes
Conn keep-alive

Thanks.

chancancode commented 12 years ago

Should we close this?

drbrain commented 12 years ago

Since this was reported before 2.5 was released I'd like to verify it first.

drbrain commented 12 years ago

I can only verify the timeout in mechanize 2.5 which is broken.

With 2.4 and 2.5.1 I can't reproduce the timeout. @mamantoha if you still experience the timeout with mechanize 2.5.1 I suspect you may have a network issue.

mamantoha commented 12 years ago

Thank you.