Chunked transfer-encoding causes exceptions at end of WARC record

[sebastian-nagel]

Reading the payload with Transfer-Encoding chunks may result in an exception thrown after the entire chunked body has been consumed.

• EOFException (http_chunked_1b.warc.gz):

  Exception in thread "main" java.io.EOFException: EOF reached before end of chunked encoding: ...ews:news>\n</url>\n</urlset>\r\n00000000\r\n\r\n<-- HERE -->

• ParseException (http_chunked_2.warc.gz):

  Exception in thread "main" org.netpreserve.jwarc.ParsingException: chunked encoding at position 1392: ...\n Total Excuted Time : 0.543\n-->\n\r\n0\r\n\r\n<-- HERE -->\r

WARC files have been recorded using Wget. See #23 for the logging of the current context (position in buffer/stream).

[sebastian-nagel]

Anybody already working on this? Otherwise I would try to fix it.

[ato]

http_chunked_1b.warc.gz

It looks like I mistranslated the BNF. RFC7230 says:

last-chunk     = 1*("0") [ chunk-ext ] CRLF

But ChunkedBody.rl has:

last_chunk = "0\r\n";

So that probably needs to be:

last_chunk = "0"+ chunk_ext* "\r\n";

in order to correctly match the zero-padded chunk length "00000000".

http_chunked_2.warc.gz

Checking it with a hex editor, it looks to me like the value of the WARC Content-Length header in the record is off by one again so the first CR from the CRLFCRLF trailer is being interpreted as part of the payload.

[sebastian-nagel]

last_chunk = "0"+ chunk_ext* "\r\n";

Yes. I've also had to check RFC 5234 to get that 1* means "at least one" and not exactly one. Otherwise 1*HEXDIG would also make no sense.

[ato]

I've created issue #29 for the content-length off by one issue seen in http_chunked_2.warc.gz.