search

erlcloud / lhttpc

lhttpc is a lightweight HTTP/1.1 client implemented in Erlang.
Other
1 stars 19 forks source link

Ignore `content-length` when TE chunked is present #4

Closed fp closed 7 years ago

fp commented 7 years ago

Some servers return BOTH transfer-encoding: chunked as well as content-length, which in error, but not much help for the client. This fixes the response handling so that it ignores the content-length header if transfer-encoding: chunked is present in the response, which makes the client "work" in the presence of such a misbehaving server.

fp commented 7 years ago

I added a test case for this function, but the _t version fail. Apparently, the handling of trailers does not work master with both Erlang 19.1 and 18.2.1, so neither doe this:

$ make test
./rebar compile
==> lhttpc (compile)
Compiled src/lhttpc_lib.erl
Compiled src/lhttpc.erl
Compiled src/lhttpc_client.erl
Compiled src/lhttpc_sup.erl
Compiled src/lhttpc_sock.erl
Compiled src/lhttpc_manager.erl
./rebar eunit
==> lhttpc (eunit)
Compiled test/lhttpc_lib_tests.erl
Compiled test/lhttpc_manager_tests.erl
Compiled test/socket_server.erl
Compiled test/webserver.erl
test/lhttpc_tests.erl:403: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
test/lhttpc_tests.erl:419: Warning: erlang:now/0: Deprecated BIF. See the "Time and Time Correction in Erlang" chapter of the ERTS User's Guide for more information.
Compiled test/lhttpc_tests.erl
test/simple_load.erl:3: Warning: behaviour gen_httpd undefined
Compiled test/simple_load.erl
Compiled src/lhttpc_lib.erl
Compiled src/lhttpc.erl
Compiled src/lhttpc_sock.erl
Compiled src/lhttpc_sup.erl
Compiled src/lhttpc_manager.erl
Compiled src/lhttpc_client.erl
======================== EUnit ========================
module 'webserver'
module 'socket_server'
module 'simple_load'
module 'lhttpc_sup'
module 'lhttpc_sock'
module 'lhttpc_manager'
  module 'lhttpc_manager_tests'
    lhttpc_manager_tests:55: manager_test_...[0.023 s] ok
    lhttpc_manager_tests:56: manager_test_...[0.001 s] ok
    lhttpc_manager_tests:57: manager_test_...[3.110 s] ok
    lhttpc_manager_tests:58: manager_test_...[16.026 s] ok
    lhttpc_manager_tests:59: manager_test_...[0.001 s] ok

=INFO REPORT==== 23-Dec-2016::11:34:50 ===
    application: lhttpc
    exited: stopped
    type: temporary

=INFO REPORT==== 23-Dec-2016::11:34:50 ===
    application: ssl
    exited: stopped
    type: temporary
    [done in 19.176 s]
  [done in 19.302 s]
module 'lhttpc_lib'
  module 'lhttpc_lib_tests'
    lhttpc_lib_tests:37: parse_url_test_...ok
    lhttpc_lib_tests:47: parse_url_test_...ok
    lhttpc_lib_tests:57: parse_url_test_...ok
    lhttpc_lib_tests:67: parse_url_test_...ok
    lhttpc_lib_tests:77: parse_url_test_...ok
    lhttpc_lib_tests:87: parse_url_test_...ok
    lhttpc_lib_tests:97: parse_url_test_...ok
    lhttpc_lib_tests:107: parse_url_test_...ok
    lhttpc_lib_tests:117: parse_url_test_...[0.028 s] ok
    lhttpc_lib_tests:128: parse_url_test_...ok
    lhttpc_lib_tests:138: parse_url_test_...ok
    lhttpc_lib_tests:148: parse_url_test_...ok
    lhttpc_lib_tests:158: parse_url_test_...ok
    lhttpc_lib_tests:168: parse_url_test_...ok
    lhttpc_lib_tests:178: parse_url_test_...ok
    lhttpc_lib_tests:188: parse_url_test_...ok
    lhttpc_lib_tests:198: parse_url_test_...ok
    lhttpc_lib_tests:208: parse_url_test_...ok
    lhttpc_lib_tests:218: parse_url_test_...ok
    [done in 0.085 s]
  [done in 0.085 s]
module 'lhttpc_client'
module 'lhttpc'
  module 'lhttpc_tests'
    lhttpc_tests:116: tcp_test_...[0.031 s] ok
    lhttpc_tests:117: tcp_test_...[0.005 s] ok
    lhttpc_tests:118: tcp_test_...[0.002 s] ok
    lhttpc_tests:119: tcp_test_...[0.005 s] ok
    lhttpc_tests:120: tcp_test_...[0.001 s] ok
    lhttpc_tests:121: tcp_test_...[0.001 s] ok
    lhttpc_tests:122: tcp_test_...[0.002 s] ok
    lhttpc_tests:123: tcp_test_...[0.002 s] ok
    lhttpc_tests:124: tcp_test_...[0.002 s] ok
    lhttpc_tests:125: tcp_test_...[0.003 s] ok
    lhttpc_tests:126: tcp_test_...[0.001 s] ok
    lhttpc_tests:127: tcp_test_...[0.002 s] ok
    lhttpc_tests:128: tcp_test_...[0.002 s] ok
    lhttpc_tests:129: tcp_test_...[0.002 s] ok
    lhttpc_tests:130: tcp_test_...[0.001 s] ok
    lhttpc_tests:131: tcp_test_...[0.001 s] ok
    lhttpc_tests:132: tcp_test_...[0.001 s] ok
    lhttpc_tests:133: tcp_test_...[0.001 s] ok
    lhttpc_tests:134: tcp_test_...[0.002 s] ok
    lhttpc_tests:135: tcp_test_...[0.001 s] ok
    lhttpc_tests:136: tcp_test_...[0.002 s] ok
    lhttpc_tests:137: tcp_test_...[0.003 s] ok
    lhttpc_tests:138: tcp_test_...[0.002 s] ok
    lhttpc_tests:139: tcp_test_...[0.003 s] ok
    lhttpc_tests:140: tcp_test_...[0.002 s] ok
    lhttpc_tests:141: tcp_test_...[0.001 s] ok
    lhttpc_tests:142: tcp_test_...ok
    lhttpc_tests:143: tcp_test_...[0.002 s] ok
    lhttpc_tests:144: tcp_test_...[0.055 s] ok
    lhttpc_tests:145: tcp_test_...[0.108 s] ok
    lhttpc_tests:146: tcp_test_...[0.056 s] ok
    lhttpc_tests:147: tcp_test_...*failed*
in function lhttpc_tests:'-chunked_encoding/0-fun-8-'/2 (test/lhttpc_tests.erl, line 500)
in call from lhttpc_tests:chunked_encoding/0 (test/lhttpc_tests.erl, line 499)
**error:{assertEqual,[{module,lhttpc_tests},
              {line,500},
              {expression,"lhttpc_lib : header_value ( \"trailer-1\" , headers ( SecondResponse ) )"},
              {expected,"1"},
              {value,undefined}]}
  output:<<"">>

    lhttpc_tests:148: tcp_test_...[0.003 s] ok
    lhttpc_tests:149: tcp_test_...[0.002 s] ok
    lhttpc_tests:150: tcp_test_...
=ERROR REPORT==== 23-Dec-2016::11:34:51 ===
Error in process <0.455.0> with exit value:
{closed,[{webserver,read_chunks,3,[{file,"test/webserver.erl"},{line,77}]},
         {webserver,read_chunked,3,[{file,"test/webserver.erl"},{line,56}]},
         {lhttpc_tests,chunked_upload,5,
                       [{file,"test/lhttpc_tests.erl"},{line,918}]},
         {webserver,server_loop,5,[{file,"test/webserver.erl"},{line,117}]},
         {webserver,accept_connection,4,
                    [{file,"test/webserver.erl"},{line,52}]}]}
*skipped*
    undefined
    *unexpected termination of test process*
::{closed,[{webserver,read_chunks,3,[{file,"test/webserver.erl"},{line,77}]},
           {webserver,read_chunked,3,[{file,"test/webserver.erl"},{line,56}]},
           {lhttpc_tests,chunked_upload,5,
                         [{file,"test/lhttpc_tests.erl"},{line,918}]},
           {webserver,server_loop,5,[{file,"test/webserver.erl"},{line,117}]},
           {webserver,accept_connection,4,[{file,[...]},{line,...}]}]}

=INFO REPORT==== 23-Dec-2016::11:34:51 ===
    application: lhttpc
    exited: stopped
    type: temporary

=INFO REPORT==== 23-Dec-2016::11:34:51 ===
    application: ssl
    exited: stopped
    type: temporary
  lhttpc_tests:172: ssl_test_...[0.238 s] ok
  lhttpc_tests:173: ssl_test_...[0.005 s] ok
  lhttpc_tests:174: ssl_test_...[0.006 s] ok
  lhttpc_tests:175: ssl_test_...[0.007 s] ok
  lhttpc_tests:176: ssl_test_...[0.052 s] ok

=INFO REPORT==== 23-Dec-2016::11:34:51 ===
    application: lhttpc
    exited: stopped
    type: temporary

=INFO REPORT==== 23-Dec-2016::11:34:51 ===
    application: ssl
    exited: stopped
    type: temporary
  lhttpc_tests:182: other_test_...ok
  [done in 0.752 s]
[done in 0.752 s]
=======================================================
  Failed: 1.  Skipped: 0.  Passed: 63.
One or more tests were cancelled.
Cover analysis: /Users/pfisher/src/ingestion/lhttpc/.eunit/index.html

=INFO REPORT==== 23-Dec-2016::11:34:51 ===
    application: asn1
    exited: stopped
    type: temporary

=INFO REPORT==== 23-Dec-2016::11:34:51 ===
    application: public_key
    exited: stopped
    type: temporary
ERROR: One or more eunit tests failed.
make: *** [test] Error 1
fp commented 7 years ago

Second commit fixes the problems with trailing headers.

fp commented 7 years ago

I stared at the remaining error in the webserver for chunked_upload, but cannot see how the socket of the test client connected to the webserver is getting closed, which results in this failure:

=ERROR REPORT==== 23-Dec-2016::14:19:48 ===
Error in process <0.381.0> with exit value:
{{closed,[]},
 [{webserver,read_chunks,3,[{file,"test/webserver.erl"},{line,77}]},
  {webserver,read_chunked,3,[{file,"test/webserver.erl"},{line,56}]},
  {lhttpc_tests,chunked_upload,5,[{file,"test/lhttpc_tests.erl"},{line,932}]},
  {webserver,server_loop,5,[{file,"test/webserver.erl"},{line,117}]},
  {webserver,accept_connection,4,[{file,"test/webserver.erl"},{line,52}]}]}
*skipped*
    undefined
    *unexpected termination of test process*
::{{closed,[]},
   [{webserver,read_chunks,3,[{file,"test/webserver.erl"},{line,77}]},
    {webserver,read_chunked,3,[{file,"test/webserver.erl"},{line,56}]},
    {lhttpc_tests,chunked_upload,5,
                  [{file,"test/lhttpc_tests.erl"},{line,932}]},
    {webserver,server_loop,5,[{file,"test/webserver.erl"},{line,117}]},
    {webserver,accept_connection,4,[{file,[...]},{line,...}]}]}

As I said, this seems to be already present on master, so apparently no one has figure it out.

motobob commented 7 years ago

already fixed in https://algithub.pd.alertlogic.net/alertlogic/lhttpc and will be merged here with #3 . i'll see if adding your new tests makes sense too