search

savonrb / httpi

Common interface for Ruby's HTTP clients
http://httpirb.com
MIT License
301 stars 151 forks source link

HTTPI is truncating response headers #69

Closed rogerleite closed 11 years ago

rogerleite commented 11 years ago

Some evidence:

$ curl -I http://br.yahoo.com
[...]
Set-Cookie: IU=deleted; expires=Mon, 28-Nov-2011 13:05:46 GMT; path=/; domain=.yahoo.com
Set-Cookie: PH=deleted; expires=Mon, 28-Nov-2011 13:05:46 GMT; path=/; domain=.yahoo.com
Set-Cookie: MSC=t=1354021547X; expires=Wed, 27-Nov-2013 13:05:47 GMT; path=/; domain=.yahoo.com
Set-Cookie: fpc=d=pT0Q_yjRlaNbj84q3kJ0Bu4REjib1FNsC106B3kSpU.P9eBX64Ml3SsgKCkjpGmQBrZYZM15huzayuXz0L27ZIv8OnCpswr0kHvK8BWZBXZ4JnfS9wHnXexE9eODzw6FlLJ34eAqNdguDn5uMxUMh2tqped8imj.Y5ghMCQ7HdBqlXwDNN1r1oOt0KXHWEkptCGBre0-&v=2; expires=Wed, 27-Nov-2013 13:05:47 GMT; path=/; domain=br.yahoo.com
Set-Cookie: fpps=deleted; expires=Mon, 28-Nov-2011 13:05:46 GMT; path=/; domain=br.yahoo.com
Set-Cookie: fpt=d=fAgQuh3XeVyQ0SZnx0E2dD4HQyzXrkNQR_Jd_4uKZKR0ewy2D_vne_SmtTyH1VlczCWGQ4gL4eAByIwWDuNYn9u7QBrSncNoDVahgD9_xUzjpSYP4OVswNAvAn4uF46W23Fq34E0bjrs1WCmmvtQ1Dpv8hFMBlRC_kf33_c0bDmLGhXDkPIs6g2OkPIhrv8OjBctFDfx.0yBvQoaO0Ms69FqoDwODKUK.j9JNHF1r_mila0e6IOjPVMytQZVpg1QDXoqqhP4fXAwdrc7ENCjibSB0PM0lkI_qIfhDHJCeyf9akU.DP1KBv2rG4lmbT0ENVx_p67pNuezqCDLoyBR0ugXbD9Wh1hiQgmp1zvYk9iHyi2ZFgeF._z_C0iZ4CXx7UdaxuOHymdES8dJlJoVrCgb7ijKjWnM7EeQgiqFxE9gNqQHsr2tRY11ThSDssQs.RKi0LxwutsFT3b5oBVFJSqOh6kTzUWJIe8ZSfGTMQ0RKRKwxks1rDqju1UfRBBoRVq5DAGSiJOzo8_MykJ8qsL19b9iJm65O5iF_gEG2YrpO3iU8XSYcwPb5tWWGymS1Q--&v=1; path=/; domain=br.yahoo.com
Set-Cookie: fpc_s=d=2zRQRMjRlaPIhaPVoWgXBdq00OJxzfvr16KeQbtQGdVmUbvECiDJkiH5LSUHZtoiSmhazOoHh_1_nTZbCDYQ5cg9uh.WRVa4fo73H6pqeGVe3BZwKSNngqkYOX4xVIs2m3aiDPOaoGqbfAxVx8hOYAFHUhcJ_oJ6T_dxqRWv.rgYkSGZJhceYFJLOl.Locpkhpg1W4XJsfrjC7ywDuFoTB3Ern4XmYP2ncrB5KWohx4IpXLKqIiMsdtLrvjt6w7UQhSpDrJXcA_gwvT9B7PKSxy2ETMK1OsY.4y.RvZpnzGc.us7.d59NSFdQMkQSbrpwIM5agg8DiofQiBxIakBp_6SzmsWAqejLButP.GMkJAms2_UixKAYko9TpLzfWeNB0GzGTeNetu4zE3Er2R6KsOugwgV7SkVsfgrdU75iiiwveMn&v=2; path=/; domain=br.yahoo.com
[...]

Using a simple script (httpi 2.0.0.rc1), i realized that only curb adapter return all "Set-Cookie" headers.

HTTPI.log = true
[:net_http, :httpclient, :curb].each do |adapter|
  request = HTTPI::Request.new("http://br.yahoo.com")
  request.headers = {"User-Agent" => "curl/7.21.4"}
  response = HTTPI.get(request, adapter)
  puts "Set-Cookie: #{response.headers["Set-Cookie"].inspect}"
end

Results:

D, [2012-11-27T11:03:16.562429 #41548] DEBUG -- : HTTPI GET request to br.yahoo.com (net_http)
Set-Cookie: "IU=deleted; expires=Mon, 28-Nov-2011 13:03:12 GMT; path=/; domain=.yahoo.com"

D, [2012-11-27T11:03:21.086947 #41548] DEBUG -- : HTTPI GET request to br.yahoo.com (httpclient)
Set-Cookie: "fpc_s=d=e.AfOJJslaP77_ZLGuCCmy9bbCmZApcoySYRckZpfsFCEFQgFVg9Z6CMxocrOtzuVAIrQuUtt28S1mqpXveqZ5ca83Kc.T7LSUbNId6CfYoxp70q7KeLbcLVgxezcpXACefDxG2AaupWVzzoLh7CZLoQk33Ei5fezikXbJMIGEpzOdF.ys0WqlJWRsolZP4LZ0oRADoIOydKwaF2VMXECDpX_A0UWa9js1CkiJ.DlGvM4sZjM18KRNzL_9uRI.MULS0ytpLfQfK0xGog2sirqj05a74Q9SCtdcA13s5Tj3PjwuAZnxxSzJnTyDwrdSscKvsiw3.UNg5oqLdQ5fgSEX7oBRaUvWWkVBVH_cvPr8RB9e.T2jJTKMUZ.tDd2E3aZ4mQ2htpHbkBEzgfMzbbKjbYIJF2avg8GvOQsRGf3SSkheeH&v=2; path=/; domain=br.yahoo.com"

D, [2012-11-27T11:03:22.969444 #41548] DEBUG -- : HTTPI GET request to br.yahoo.com (curb)
Set-Cookie: ["IU=deleted; expires=Mon, 28-Nov-2011 13:03:19 GMT; path=/; domain=.yahoo.com", "PH=deleted; expires=Mon, 28-Nov-2011 13:03:19 GMT; path=/; domain=.yahoo.com", "MSC=t=1354021400X; expires=Wed, 27-Nov-2013 13:03:20 GMT; path=/; domain=.yahoo.com", "fpc=d=twmBb_tilaM2g8UhO2xUJ267pR5WJEyQjQBCKZ6P99Py8EcPbZnnJYuWtk6.dDsQcOtpR6NXZ8_n37uA2hmXRRyS60QoSnbBh4llNfxyPNleDwnFykzKyE0JRevgDtN8Wc3N_Rx9I4QZb9r9aVwkfaKEYJdlXxUpe37gqYuoHmiuWuzze1AkdL96T3icYn_LQJACNtc-&v=2; expires=Wed, 27-Nov-2013 13:03:20 GMT; path=/; domain=br.yahoo.com", "fpps=deleted; expires=Mon, 28-Nov-2011 13:03:19 GMT; path=/; domain=br.yahoo.com", "fpt=d=cgP9GTbXeVzKcBBu441lu6snPn5G0OxnEsu2HHFRGns2MWR.hszVlrrBI4SsPg.oknioT7TkHxn.hYhTwUwn..8CnrM7pu4iYnxuV6MgpPniQAQbb2glPUjTcr1z.CbH8RQA42th7C9L0gVOQDltnmxPv_2OFp76xDs.MpAoaqAVPbXClozyDh6tRw4CCllehQw2f7TQsFxHyi9I50sTyNOAmUcOpejEP1jJjAZfY.F5.h9poAtTiov.Yz.8UjELZSVNZKgn6vnDw9VmQUUTEI45iqiLGCaZSlgzxv1pbOh7KqihVY5itiWBMIVZn51X0huBWDv07bZjXxgXAVHel5i9QBzGBfUw0AmDPn2e56dT44_lvrK5U1WpNSkjcOU2nVZ_3wkf0ClKEg8AymM3k5ZDtVd4SIT5VSv8c332gLFvY4VKEMkH1et.A6SWH79TtkXVoPViOvJT978GWiEKJFpU1H7W3kzAtfeUKVnZnzMQ7udTVXON9_Mxhyu.B0pZyysv4mEc.rXLjOoF7CU1J7rP3T6MNZBYQwAxgH2qkLPoNecOmlTmfNsgVXjRXAgQLw--&v=1; path=/; domain=br.yahoo.com", "fpc_s=d=dcozG8hilaMbWa30O6JWUesgDD1W_k_SBg3VMdMWl3S0sFfGT5hPgeGEpvStnJj1BKsXkzQhhdn0gx7L2XN79Bu5SMBVb4.xrtHR.x6YMgRib4B7MPsR3l4rL8dS_EAeTJmGOeadUph6WQeZokCsb3t66iOVnZ3fxfDt_7KkS9EfiJMblqYywobLceiOhIjwgKfFVmm0XsyahY_PDWmD6MyjgcCJQPmM7QVhP_mMWB5rSNjSHI3KzP24COyBbGgrdNQE4MzrFu8xL.Q7TDiCMZcCZec352gsvVXTK_y59W7b9F73Fhq4d_sfFLHh4nSDbixXEYN__Tv2O8Hl7.HA9QR.jbP2iO1HIELslzXg1cAOxPkXpelfEhLsWQJsTRp0rJwFMy5JrkvTwLjWgdAR6soo7w_1FFgLu.vEmbGA.eKjy3cO&v=2; path=/; domain=br.yahoo.com"]

I'm planning to work on net_http and httpclient adapters to return headers like curb do. Is that ok?

rogerleite commented 11 years ago

i can't simulate this error because webmock's magic. See this commit for example: https://github.com/bblimke/webmock/commit/553496851cebbef505708c91f99ae1d00151a9d6

@rubiii Any idea? I started to work on this branch: https://github.com/rogerleite/httpi/tree/fix_response_headers

rubiii commented 11 years ago

no idea right now, but i'll take a look at it later.

rubiii commented 11 years ago

thanks @rogerleite, but i need specs for this. i'll look into getting rid of webmock. i was planning to replace it with a simple server anyway.

rogerleite commented 11 years ago

@rubiii i kind of extracted MinionServer to help with it. I'm using on integration tests of http_monkey. If you want to help, i can add you as collaborator. What do you think?

rogerleite commented 11 years ago

I need this for http_monkey, i can try to simulate this bug on integration tests. Ok with that?

rubiii commented 11 years ago

have to take a look at this "after work" ;) i kind of started something similar here. don't know if we're aiming for the same tool here.

rogerleite commented 11 years ago

Your IntegrationServer is really cool. The main goal is to provide a integration server that supports "rack app" and i can control the start and stop. You server is cool because also supports SSL. Many projects needs this, and always use some kind of "home made" solution.

rubiii commented 11 years ago

yes, the idea was to have a basic server that can run rack apps (just like your minion server) and add n applications for every kind of http test case one can imagine (ssl, redirects, errors, timeouts, cookies, etc).

rogerleite commented 11 years ago

Great! The invite as collaborator is still open! :smile:

rubiii commented 11 years ago

thanks :) i would really like to hear what your plans are for minion server and whether we can combine efforts. but this issue may not be the right place for that ;)

rubiii commented 11 years ago

@rogerleite i removed webmock from master. hope this solves your problem.

rogerleite commented 11 years ago

Thanks @rubiii I'll write some tests for this issue!