search

cmullaparthi / ibrowse

Erlang HTTP client
Other
516 stars 190 forks source link

Strem next after request process timeout #86

Closed nevar closed 10 years ago

nevar commented 11 years ago 
347> f(ReqID), {ibrowse_req_id, ReqID} = ibrowse:send_req("http://www.google.ru/", [], get, [], [{stream_chunk_size, 1}, {stream_to, {self(), once}}, {response_format, binary}, {trace, true}]).
2013-2-22_5:31:20:405 -- (www.google.ru:80) - Connecting...
2013-2-22_5:31:20:455 -- (www.google.ru:80) - Connected! Socket: #Port<0.22539>
2013-2-22_5:31:20:456 -- (www.google.ru:80) - Sending request: 
--- Request Begin ---
GET / HTTP/1.1
Host: www.google.ru
Content-Length: 0

--- Request End ---
2013-2-22_5:31:20:456 -- (www.google.ru:80) - Sending request body: 
--- Request Body Begin ---

--- Request Body End ---
{ibrowse_req_id,{1361,485880,456002}}
348> 2013-2-22_5:31:20:528 -- (www.google.ru:80) - Data recvd in state: get_header. Size: 1351. <<"HTTP/1.1 200 OK\r\nDate: Fri, 22 Feb 2013 06:31:37 GMT\r\nExpires: -1\r\nCache-Control: private, max-age=0\r\nContent-Type: text/html; charset=windows-1251\r\nSet-Cookie: PREF=ID=fff1d8bbc49d365f:FF=0:NW=1:TM=1361514697:LM=1361514697:S=6sEgw0wGi57NiFTw; expires=Sun, 22-Feb-2015 06:31:37 GMT; path=/; domain=.google.ru\r\nSet-Cookie: NID=67=qJ_xnefrJJpNoGTUTPC5J7j07u2OLw-ZNLQVQEvqup1SPqUCfRJo3S7TcF_YMSCZdjBywLmj6Uf9WLZPmHcxPAHEQETgdpXeHeKONOybXvc9XojUpl0KI5BgaPZBjN57; expires=Sat, 24-Aug-2013 06:31:37 GMT; path=/; domain=.google.ru; HttpOnly\r\nP3P: CP=\"This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info.\"\r\nServer: gws\r\nX-XSS-Protection: 1; mode=block\r\nX-Frame-Options: SAMEORIGIN\r\nTransfer-Encoding: chunked\r\n\r\n23f\r\n<!doctype html><html itemscope=\"itemscope\" itemtype=\"http://schema.org/WebPage\"><head><meta content=\"\317\356\350\361\352 \350\355\364\356\360\354\340\366\350\350 \342 \350\355\362\345\360\355\345\362\345: \342\345\341 \361\362\360\340\355\350\366\373, \352\340\360\362\350\355\352\350, \342\350\344\345\356 \350 \354\355\356\343\356\345 \344\360\363\343\356\345.\" name=\"description\"><meta content=\"noodp\" name=\"robots\"><meta itemprop=\"image\" content=\"/images/google_favicon_128.png\"><title>Google</title><script>(function(){\nwindow.google={kEI:\"yRAnUfO7O6SE4gT004DQDg\",getEI:function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute(\"eid\")));)a=a.parentNode;return b||google.kEI},https:function(){return\"https:\"==window.location.protocol},kEXPI:\"17259,3952\r\n">>

2013-2-22_5:31:20:529 -- (www.google.ru:80) - Recvd Header Data -> HTTP/1.1 200 OK
Date: Fri, 22 Feb 2013 06:31:37 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=windows-1251
Set-Cookie: PREF=ID=fff1d8bbc49d365f:FF=0:NW=1:TM=1361514697:LM=1361514697:S=6sEgw0wGi57NiFTw; expires=Sun, 22-Feb-2015 06:31:37 GMT; path=/; domain=.google.ru
Set-Cookie: NID=67=qJ_xnefrJJpNoGTUTPC5J7j07u2OLw-ZNLQVQEvqup1SPqUCfRJo3S7TcF_YMSCZdjBywLmj6Uf9WLZPmHcxPAHEQETgdpXeHeKONOybXvc9XojUpl0KI5BgaPZBjN57; expires=Sat, 24-Aug-2013 06:31:37 GMT; path=/; domain=.google.ru; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Transfer-Encoding: chunked
----
2013-2-22_5:31:20:529 -- (www.google.ru:80) - Recvd headers
--- Headers Begin ---
HTTP/1.1 200 OK
Date: Fri, 22 Feb 2013 06:31:37 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=windows-1251
Set-Cookie: PREF=ID=fff1d8bbc49d365f:FF=0:NW=1:TM=1361514697:LM=1361514697:S=6sEgw0wGi57NiFTw; expires=Sun, 22-Feb-2015 06:31:37 GMT; path=/; domain=.google.ru
Set-Cookie: NID=67=qJ_xnefrJJpNoGTUTPC5J7j07u2OLw-ZNLQVQEvqup1SPqUCfRJo3S7TcF_YMSCZdjBywLmj6Uf9WLZPmHcxPAHEQETgdpXeHeKONOybXvc9XojUpl0KI5BgaPZBjN57; expires=Sat, 24-Aug-2013 06:31:37 GMT; path=/; domain=.google.ru; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Transfer-Encoding: chunked
--- Headers End ---

2013-2-22_5:31:20:530 -- (www.google.ru:80) - HttpVsn: "HTTP/1.1" StatusCode: "200" Headers_1 -> [{"Date","Fri, 22 Feb 2013 06:31:37 GMT"},{"Expires","-1"},{"Cache-Control","private, max-age=0"},{"Content-Type","text/html; charset=windows-1251"},{"Set-Cookie","PREF=ID=fff1d8bbc49d365f:FF=0:NW=1:TM=1361514697:LM=1361514697:S=6sEgw0wGi57NiFTw; expires=Sun, 22-Feb-2015 06:31:37 GMT; path=/; domain=.google.ru"},{"Set-Cookie","NID=67=qJ_xnefrJJpNoGTUTPC5J7j07u2OLw-ZNLQVQEvqup1SPqUCfRJo3S7TcF_YMSCZdjBywLmj6Uf9WLZPmHcxPAHEQETgdpXeHeKONOybXvc9XojUpl0KI5BgaPZBjN57; expires=Sat, 24-Aug-2013 06:31:37 GMT; path=/; domain=.google.ru; HttpOnly"},{"P3P","CP=\"This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info.\""},{"Server","gws"},{"X-XSS-Protection","1; mode=block"},{"X-Frame-Options","SAMEORIGIN"},{"Transfer-Encoding","chunked"}]
2013-2-22_5:31:20:530 -- (www.google.ru:80) - Chunked encoding detected...
2013-2-22_5:31:20:530 -- (www.google.ru:80) - Determined chunk size: 575. Already recvd: 577
2013-2-22_5:31:20:530 -- (www.google.ru:80) - Recvd more data: size: 577. NeedBytes: 575
2013-2-22_5:31:20:530 -- (www.google.ru:80) - Recvd another chunk...<<"<!doctype html><html itemscope=\"itemscope\" itemtype=\"http://schema.org/WebPage\"><head><meta content=\"\317\356\350\361\352 \350\355\364\356\360\354\340\366\350\350 \342 \350\355\362\345\360\355\345\362\345: \342\345\341 \361\362\360\340\355\350\366\373, \352\340\360\362\350\355\352\350, \342\350\344\345\356 \350 \354\355\356\343\356\345 \344\360\363\343\356\345.\" name=\"description\"><meta content=\"noodp\" name=\"robots\"><meta itemprop=\"image\" content=\"/images/google_favicon_128.png\"><title>Google</title><script>(function(){\nwindow.google={kEI:\"yRAnUfO7O6SE4gT004DQDg\",getEI:function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute(\"eid\")));)a=a.parentNode;return b||google.kEI},https:function(){return\"https:\"==window.location.protocol},kEXPI:\"17259,3952">>
2013-2-22_5:31:20:531 -- (www.google.ru:80) - RemData -> <<"\r\n">>
348> flush().                                                                                                                                                             Shell got {ibrowse_async_headers,{1361,485880,456002},
                                 "200",
                                 [{"Date","Fri, 22 Feb 2013 06:31:37 GMT"},
                                  {"Expires","-1"},
                                  {"Cache-Control","private, max-age=0"},
                                  {"Content-Type",
                                   "text/html; charset=windows-1251"},
                                  {"Set-Cookie",
                                   "PREF=ID=fff1d8bbc49d365f:FF=0:NW=1:TM=1361514697:LM=1361514697:S=6sEgw0wGi57NiFTw; expires=Sun, 22-Feb-2015 06:31:37 GMT; path=/; domain=.google.ru"},
                                  {"Set-Cookie",
                                   "NID=67=qJ_xnefrJJpNoGTUTPC5J7j07u2OLw-ZNLQVQEvqup1SPqUCfRJo3S7TcF_YMSCZdjBywLmj6Uf9WLZPmHcxPAHEQETgdpXeHeKONOybXvc9XojUpl0KI5BgaPZBjN57; expires=Sat, 24-Aug-2013 06:31:37 GMT; path=/; domain=.google.ru; HttpOnly"},
                                  {"P3P",
                                   "CP=\"This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info.\""},
                                  {"Server","gws"},
                                  {"X-XSS-Protection","1; mode=block"},
                                  {"X-Frame-Options","SAMEORIGIN"},
                                  {"Transfer-Encoding","chunked"}]}
Shell got {ibrowse_async_response,{1361,485880,456002},
                                  <<"<!doctype html><html itemscope=\"itemscope\" itemtype=\"http://schema.org/WebPage\"><head><meta content=\"\317\356\350\361\352 \350\355\364\356\360\354\340\366\350\350 \342 \350\355\362\345\360\355\345\362\345: \342\345\341 \361\362\360\340\355\350\366\373, \352\340\360\362\350\355\352\350, \342\350\344\345\356 \350 \354\355\356\343\356\345 \344\360\363\343\356\345.\" name=\"description\"><meta content=\"noodp\" name=\"robots\"><meta itemprop=\"image\" content=\"/images/google_favicon_128.png\"><title>Google</title><script>(function(){\nwindow.google={kEI:\"yRAnUfO7O6SE4gT004DQDg\",getEI:function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute(\"eid\")));)a=a.parentNode;return b||google.kEI},https:function(){return\"https:\"==window.location.protocol},kEXPI:\"17259,3952">>}
ok
349> timer:sleep(30000).
ok
350> flush().           
ok
351> ibrowse:stream_next(ReqID).                                                                                                                                          ok                     
352> flush().                   
ok
353>

Is it right behaviour? I expect receive error like:

355> ibrowse:send_req("http://www.google.ru/", [], get, [], [{stream_chunk_size, 1}, {stream_to, {self(), once}}, {response_format, binary}, {trace, true}], 5). 
2013-2-22_5:34:47:725 -- (www.google.ru:80) - Connecting...
{error,req_timedout}
356> 2013-2-22_5:34:47:733 -- (www.google.ru:80) - Error connecting. Reason: {error,timeout}
cmullaparthi commented 11 years ago

Thanks for reporting this. I am looking into it.

cmullaparthi commented 10 years ago

Fixed in 4.1.0