Content-Length: 0 in Mojo::UserAgent get

[fayland]

• Mojolicious version: 8.33
• Perl version: 5.22.1 (guess all)
• Operating system: Ubuntu 16 and MacOSx

Steps to reproduce the behavior

MOJO_CLIENT_DEBUG=1 perl -MMojo::UserAgent -E "say Mojo::UserAgent->new->get('https://wtfismyip.com/text');" -- Blocking request (https://wtfismyip.com/text) -- Connect 5927b731f3f3b60f53b78941304a40af (https://wtfismyip.com:443) -- Client >>> Server (https://wtfismyip.com/text) GET /text HTTP/1.1\x0d Content-Length: 0\x0d Host: wtfismyip.com\x0d Accept-Encoding: gzip\x0d User-Agent: Mojolicious (Perl)\x0d \x0d

Expected behavior

We should not have "Content-Length: 0" on the request.

Actual behavior

it has "Content-Length: 0" inside. actually I have problem for one site when I send "Content-Length: 0" in the header.

[fayland]

it's in Mojo::Message

sub fix_headers {
  my $self = shift;
  return $self if $self->{fix}++;

  # Content-Length or Connection (unless chunked transfer encoding is used)
  my $content = $self->content;
  my $headers = $content->headers;
  if ($content->is_multipart) { $headers->remove('Content-Length') }
  elsif ($content->is_chunked || $headers->content_length) { return $self }
  if   ($content->is_dynamic) { $headers->connection('close') }
  else                        { $headers->content_length($self->body_size) }

  return $self;
}

probably before set the $headers->content_length, it should be elsif ($self->body_size)

[Tekki]

"Any Content-Length greater than or equal to zero is a valid value." https://www.w3.org/Protocols/HTTP/1.0/spec.html#Content-Length

[fayland]

should not have Content-Length in the header for GET I think. if you do curl someurl, it won't add Content-Length in request headers as well. why we would have Content-Length: 0 in mojo? any reason? it can be fixed by elsif ($self->body_size) and it's sane I think.

[kraih]

This is most definitely not a bug.

[fayland]

why we would send Content-Length: 0 on GET? like Chrome, if you visit a url and check the network tab, in Request, it won't have Content-Length. well, why it bothers me a bit is the website betway.com, which is returning server error when we send Content-Length: 0 on GET. and we can't remove it with code actually.

Thanks

[s1037989]

@fayland would you be able to render your response with a 204 status code? That will remove the Content-Length header if it's 0 length.

Mojo::Message::Response#fix_headers:

  # RFC 7230 3.3.2
  $headers->remove('Content-Length') if $self->is_empty;
  # is_empty: Check if this response has a 1xx, 204 or 304 status "code".

Server with a 204 response:

$ perl -Mojo -E 'a("/" => sub { shift->render(text => "", status => 204); })->start' daemon

Client:

$ MOJO_CLIENT_DEBUG=1 perl -Mojo -E 'say g("http://localhost:3000/")'
-- Client <<< Server (http://localhost:3000/)
HTTP/1.1 204 No Content\x0d
Content-Type: text/html;charset=UTF-8\x0d
Date: Mon, 09 Mar 2020 02:35:55 GMT\x0d
Server: Mojolicious (Perl)\x0d
\x0d

[fayland]

it's not about render. actually I'm trying to get the data from other website as scraper.

[kraih]

You could convince me to change this by selling it as a performance optimization. But a PR would have to show benchmark results.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It may be closed if no further activity occurs. Thank you for your contributions.

[kraih]

Still nobody has done any benchmarks. 😞