search

croservices / cro-http

HTTP (including HTTPS and HTTP/2) support for the Cro library for building distributed systems in Raku.
https://cro.services/
Artistic License 2.0
49 stars 26 forks source link

Content-Length header is not being set on POST requests. #73

Open Xliff opened 5 years ago

Xliff commented 5 years ago

I have the following post request captured from using CRO_TRACE=1

[TRACE(anon 1)] RequestSerializerExtension EMIT HTTP Request
  POST /oauth/token HTTP/1.1
  Host: login.eveonline.com
  User-agent: WebService::EveOnline v0.5.0 (rakudo)
  content-type: application/x-www-form-urlencoded
  Authorization: Basic *<basic auth hash>*

This reqeust has the following payload:

[TRACE(anon 1)] Cro::ConnectionConditional EMIT TCP Message
  00 00 64 00 00 00 00 00 01 63 6f 64 65 3d 65 6d  ..d......code=em
  72 66 46 6f 46 4b 71 37 6b 49 6c 37 44 4e 46 64  rfFoFKq7kIl7DNFd
  38 6e 64 53 6a 30 77 73 41 5a 4a 4a 31 6e 5f 64  8ndSj0wsAZJJ1n_d
  6b 69 37 6b 47 62 4f 71 61 6f 52 74 5a 57 41 6e  ki7kGbOqaoRtZWAn
  4f 5a 43 45 6e 46 62 59 7a 36 73 62 45 6b 30 26  OZCEnFbYz6sbEk0&
  67 72 61 6e 74 5f 74 79 70 65 3d 61 75 74 68 6f  grant_type=autho
  72 69 7a 61 74 69 6f 6e 5f 63 6f 64 65           rization_code

The requet fails because the Content-Length header is not present. Is there something special I need to do to get CRO::HTTP::Client to send it?

Thanks

Altai-man commented 5 years ago

Can you please post at least a piece of your code that does the request? Of course, with creds and stuff changed.

Xliff commented 5 years ago

Sure.

       method refreshToken($tokenCode) {
                # POST CALL USES THIS FORM DATA
        my $form_data = {
                  grant_type      => 'authorization_code',
                  code            => $tokenCode
        }
        my $response = self.getBearerToken($form_data);

        die "Token not refreshed due to unexpected error."
            unless self.is-success($response);

        # cw: Maybe add code to output response if a flag is set?
        die "Invalid response content-type."
            unless $response.field('Content-Type') ~~ /^ 'application/json' /;

        #my $jsonObj = from-json(await $response.body);
        my $jsonObj = await $response.body;
        self.setTokenData($jsonObj);
    }

        # Protected
    method getBearerToken($form_data) {
                # POST REQUEST MADE HERE.
        await $!client.post(
                "{ EVE_SSO_PREFIX }/oauth/token",
            content-type    => 'application/x-www-form-urlencoded',
                body          => $form_data,
            auth          => {
                username => %!privateData{$!section}<client_id>,
                password => %!privateData{$!section}<secret_id>
            }
      );
    }
Altai-man commented 5 years ago

Confirmed... Fixing.

Altai-man commented 5 years ago

@Xliff Can you please post a full CRO_TRACE output?

It is true that at RequestSerializerExtension level content-length is not yet set, but in my small example it is set by RequestSerializer component, so in RequestParser part of the log you are likely to see Content-length.

Xliff commented 5 years ago

Output with full CRO_TRACE enabled.

[TRACE(anon 1)] Cro::ConnectionConditional EMIT TCP Message
  50 52 49 20 2a 20 48 54 54 50 2f 32 2e 30 0d 0a  PRI * HTTP/2.0..
  0d 0a 53 4d 0d 0a 0d 0a                          ..SM....

[TRACE(anon 1)] Cro::ConnectionConditional EMIT TCP Message
  00 00 24 04 00 00 00 00 00 00 01 00 00 10 00 00  ..$.............
  02 00 00 00 00 00 03 00 00 00 64 00 04 00 00 ff  ..........d.....
  ff 00 05 00 00 40 00 00 06 00 00 03 e8           .....@.......

[TRACE(anon 1)] RequestSerializerExtension EMIT HTTP Request
  POST /oauth/token HTTP/1.1
  Host: login.eveonline.com
  User-agent: WebService::EveOnline v0.5.0 (rakudo)
  Authorization: Basic MTI0OGY0OGVhZGJmNDc4ZGJmZGFjYjY3MTQzYTZmMWY6dnd1U25uTVZjYkxkUE5remExbjYzbkdqd1hBcTlXcGNrMEk1OGh5aGw=
  Content-Type: application/json
[TRACE(anon 1)] Cro::ConnectionConditional EMIT TCP Message
  00 00 ca 01 04 00 00 00 01 83 87 44 0c 2f 6f 61  ...........D./oa
  75 74 68 2f 74 6f 6b 65 6e 66 13 6c 6f 67 69 6e  uth/tokenf.login
  2e 65 76 65 6f 6e 6c 69 6e 65 2e 63 6f 6d 7a 25  .eveonline.comz%
  57 65 62 53 65 72 76 69 63 65 3a 3a 45 76 65 4f  WebService::EveO
  6e 6c 69 6e 65 20 76 30 2e 35 2e 30 20 28 72 61  nline v0.5.0 (ra
  6b 75 64 6f 29 57 6a 42 61 73 69 63 20 4d 54 49  kudo)WjBasic MTI
  30 4f 47 59 30 4f 47 56 68 5a 47 4a 6d 4e 44 63  0OGY0OGVhZGJmNDc
  34 5a 47 4a 6d 5a 47 46 6a 59 6a 59 33 4d 54 51  4ZGJmZGFjYjY3MTQ
  7a 59 54 5a 6d 4d 57 59 36 64 6e 64 31 55 32 35  zYTZmMWY6dnd1U25
  75 54 56 5a 6a 59 6b 78 6b 55 45 35 72 65 6d 45  uTVZjYkxkUE5remE
  78 62 6a 59 7a 62 6b 64 71 64 31 68 42 63 54 6c  xbjYzbkdqd1hBcTl
  58 63 47 4e 72 4d 45 6b 31 4f 47 68 35 61 47 77  XcGNrMEk1OGh5aGw
  3d 5f 10 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6a  =_.application/j
  73 6f 6e                                         son

[TRACE(anon 1)] Cro::ConnectionConditional EMIT TCP Message
  00 00 70 00 00 00 00 00 01 7b 22 63 6f 64 65 22  ..p......{"code"
  3a 20 22 72 66 4b 6a 64 51 4f 39 7a 7a 51 63 58  : "rfKjdQO9zzQcX
  41 41 66 53 4e 77 36 55 48 32 35 4b 38 7a 37 67  AAfSNw6UH25K8z7g
  58 64 6c 33 5a 30 73 4d 30 71 6c 37 48 66 56 6f  Xdl3Z0sM0ql7HfVo
  48 38 67 56 5a 6e 78 55 6f 54 68 68 48 51 53 43  H8gVZnxUoThhHQSC
  6a 78 37 30 22 2c 22 67 72 61 6e 74 5f 74 79 70  jx70","grant_typ
  65 22 3a 20 22 61 75 74 68 6f 72 69 7a 61 74 69  e": "authorizati
  6f 6e 5f 63 6f 64 65 22 7d                       on_code"}

[TRACE(anon 1)] Cro::TLS::Connector EMIT TCP Message
  00 00 12 04 00 00 00 00 00 00 03 00 00 00 0a 00  ................
  04 00 00 80 00 00 06 00 00 80 00                 ...........

[TRACE(anon 1)] Cro::TLS::Connector EMIT TCP Message
  00 00 00 04 01 00 00 00 00                       .........

[TRACE(anon 1)] Cro::ConnectionConditional EMIT TCP Message
  00 00 24 04 00 00 00 00 00 00 01 00 00 10 00 00  ..$.............
  02 00 00 00 00 00 03 00 00 00 64 00 04 00 00 ff  ..........d.....
  ff 00 05 00 00 40 00 00 06 00 00 03 e8           .....@.......

[TRACE(anon 1)] Cro::ConnectionConditional EMIT TCP Message
  00 00 00 04 01 00 00 00 00                       .........

[TRACE(anon 1)] Cro::ConnectionConditional EMIT TCP Message
  00 00 00 00 01 00 00 00 01                       .........

[TRACE(anon 1)] Cro::TLS::Connector EMIT TCP Message
  00 00 5b 01 04 00 00 00 01 3f e1 1f 08 82 68 21  ..[......?....h!
  5f 93 49 7c a5 89 d3 4d 1f 6a 12 71 d8 82 a6 0b  _.I|...M.j.q....
  50 b0 d0 43 1b 76 90 d0 62 58 74 1e 54 ad 8f 7e  P..C.v..bXt.T..~
  fd 70 eb c8 c0 97 07 0f 12 96 df 3d bf 4a 01 d5  .p.........=.J..
  34 0e c5 04 00 be a0 5b b8 27 ae 36 d2 98 b4 6f  4......[.'.6...o
  40 87 f2 b1 2a 29 12 63 d5 84 25 07 41 7f 0f 0d  @...*).c..%.A...
  03 33 34 34                                      .344

[TRACE(anon 1)] Cro::TLS::Connector EMIT TCP Message
  00 01 58 00 01 00 00 00 01 3c 21 44 4f 43 54 59  ..X......<!DOCTY
  50 45 20 48 54 4d 4c 20 50 55 42 4c 49 43 20 22  PE HTML PUBLIC "
  2d 2f 2f 57 33 43 2f 2f 44 54 44 20 48 54 4d 4c  -//W3C//DTD HTML
  20 34 2e 30 31 2f 2f 45 4e 22 22 68 74 74 70 3a   4.01//EN""http:
  2f 2f 77 77 77 2e 77 33 2e 6f 72 67 2f 54 52 2f  //www.w3.org/TR/
  68 74 6d 6c 34 2f 73 74 72 69 63 74 2e 64 74 64  html4/strict.dtd
  22 3e 0d 0a 3c 48 54 4d 4c 3e 3c 48 45 41 44 3e  ">..<HTML><HEAD>
  3c 54 49 54 4c 45 3e 4c 65 6e 67 74 68 20 52 65  <TITLE>Length Re
  71 75 69 72 65 64 3c 2f 54 49 54 4c 45 3e 0d 0a  quired</TITLE>..
  3c 4d 45 54 41 20 48 54 54 50 2d 45 51 55 49 56  <META HTTP-EQUIV
  3d 22 43 6f 6e 74 65 6e 74 2d 54 79 70 65 22 20  ="Content-Type" 
  43 6f 6e 74 65 6e 74 3d 22 74 65 78 74 2f 68 74  Content="text/ht
  6d 6c 3b 20 63 68 61 72 73 65 74 3d 75 73 2d 61  ml; charset=us-a
  73 63 69 69 22 3e 3c 2f 48 45 41 44 3e 0d 0a 3c  scii"></HEAD>..<
  42 4f 44 59 3e 3c 68 32 3e 4c 65 6e 67 74 68 20  BODY><h2>Length 
  52 65 71 75 69 72 65 64 3c 2f 68 32 3e 0d 0a 3c  Required</h2>..<
  68 72 3e 3c 70 3e 48 54 54 50 20 45 72 72 6f 72  hr><p>HTTP Error
  20 34 31 31 2e 20 54 68 65 20 72 65 71 75 65 73   411. The reques
  74 20 6d 75 73 74 20 62 65 20 63 68 75 6e 6b 65  t must be chunke
  64 20 6f 72 20 68 61 76 65 20 61 20 63 6f 6e 74  d or have a cont
  65 6e 74 20 6c 65 6e 67 74 68 2e 3c 2f 70 3e 0d  ent length.</p>.
  0a 3c 2f 42 4f 44 59 3e 3c 2f 48 54 4d 4c 3e 0d  .</BODY></HTML>.
  0a                                               .

[TRACE(anon 1)] Cro::ConnectionConditional EMIT HTTP Response
  HTTP/2.0 411 Length Required
  content-type: text/html; charset=us-ascii
  server: Microsoft-HTTPAPI/2.0
  date: Thu, 07 Mar 2019 15:28:54 GMT
  x-cnection: close
  content-length: 344
[TRACE(anon 1)] ResponseParserExtension EMIT HTTP Response
  HTTP/2.0 411 Length Required
  content-type: text/html; charset=us-ascii
  server: Microsoft-HTTPAPI/2.0
  date: Thu, 07 Mar 2019 15:28:54 GMT
  x-cnection: close
  content-length: 344
[TRACE(anon 1)] Cro::TLS::Connector EMIT TCP Message
  00 00 04 03 00 00 00 00 01 00 00 00 05           .............

[TRACE(anon 1)] Cro::TLS::Connector EMIT TCP Message
  00 00 08 07 00 00 00 00 00 00 00 00 01 00 00 00  ................
  00                                               .

[TRACE(anon 1)] Cro::TLS::Connector EMIT TCP Message
  00 00 00 04 01 00 00 00 00                       .........

[TRACE(anon 1)] Cro::TLS::Connector QUIT connection reset by peer

[TRACE(anon 1)] Cro::ConnectionConditional QUIT connection reset by peer

[TRACE(anon 1)] ResponseParserExtension QUIT connection reset by peer

An operation first awaited:
  in method getBearerToken at /home/cbwood/Projects/p6-webservice-eveonline/lib/WebService/EveOnline/SSO/Base.pm6 (WebService::EveOnline::SSO::Base) line 115
  in method refreshToken at /home/cbwood/Projects/p6-webservice-eveonline/lib/WebService/EveOnline/SSO/Base.pm6 (WebService::EveOnline::SSO::Base) line 150
  in block  at /home/cbwood/Projects/p6-webservice-eveonline/lib/WebService/EveOnline/SSO/Web.pm6 (WebService::EveOnline::SSO::Web) line 53
  in block  at /home/cbwood/Projects/p6-webservice-eveonline/lib/WebService/EveOnline/SSO/Web.pm6 (WebService::EveOnline::SSO::Web) line 47
  in block  at /home/cbwood/Projects/p6-webservice-eveonline/../p6-GtkPlus/lib/GTK/Roles/Signals/Generic.pm6 (GTK::Roles::Signals::Generic) line 219
  in method CALL-ME at /home/cbwood/Projects/rakudobrew/moar-master/install/share/perl6/sources/947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 587
  in method run at /home/cbwood/Projects/p6-webservice-eveonline/../p6-GtkPlus/lib/GTK/Application.pm6 (GTK::Application) line 159
  in submethod BUILD at /home/cbwood/Projects/p6-webservice-eveonline/lib/WebService/EveOnline/SSO/Web.pm6 (WebService::EveOnline::SSO::Web) line 66
  in method new at /home/cbwood/Projects/p6-webservice-eveonline/lib/WebService/EveOnline/SSO/Web.pm6 (WebService::EveOnline::SSO::Web) line 77
  in sub MAIN at bin/assetSearch.pl6 line 298
  in block <unit> at bin/assetSearch.pl6 line 539

Died with the exception:
    Server responded with 411 Length Required
      in block  at /home/cbwood/Projects/rakudobrew/moar-master/install/share/perl6/site/sources/99126695DF75E3226F5852E330363C9BF3B26F44 (Cro::HTTP::Client) line 418
Altai-man commented 5 years ago

Why do you have content-type: application/x-www-form-urlencoded and content-type => 'application/x-www-form-urlencoded', in your code, but in your latest gist there is: Content-Type: application/json? Or it doesn't matter here?

Xliff commented 5 years ago

Doesn't matter.

Altai-man commented 5 years ago

Ok, I'll try it again later. Thanks.

Altai-man commented 5 years ago

I still cannot reproduce it with golfing. I've clonned Webservice-EveOnline, but not sure how to run it and what can be needed. Is it possible to mock web parts and cut out only Cro-related pieces with the bug reproducible?

Xliff commented 5 years ago

Unfortunately, you'd have to clone more than that to reproduce the bug, and the isolated version of the code no longer works without WebKitGTK and GTKPlus.

All I can tell you is that the content length isn't getting set for specific requests.

Here is how I managed a work-around, if that helps:

my $body = to-json($form_data, :!pretty);
await $!client.post(
  "{ EVE_SSO_PREFIX }/oauth/token",
  headers        => [ 
        Content-Type     => 'application/json', 
        Content-Length => $body.chars 
  ],
  body           => $body,
  auth           => {
    username => %!privateData{$!section}<client_id>,
    password => %!privateData{$!section}<secret_id>
  }
);

Without specifying the Content-Length in the above call, it will NOT work, and will throw a 411. Sorry I can't do anything further to illustrate the issue.

sthiriet commented 2 years ago

I can reproduce with that code : 

use Cro::HTTP::Client;

my $user='xxxxxx';
my $password='yyyyyyy';

my $client = Cro::HTTP::Client.new(
    base-uri => "https://my-service",
    ca => {ca-file => '/etc/ssl/certs/ca-certificates.crt'});

my $body = { grant_type => 'client_credentials' };
my $resp = await $client.post: '/v2/token',
    content-type => 'application/x-www-form-urlencoded',
    auth => {
        username => $user,
        password => $password
    },
    body => $body,
    headers => [
        Accept  => 'application/json;charset=utf-8'
    ];

Without setting Content-length, i get a HTTP 400 : Parameter grant_type is mandatory. With @Xliff's work-around, I can get a token.