[WARC writer / protocol-okhttp] WARC-Truncated header issues and improvements

sebastian-nagel commented 5 years ago

There are some oddities how truncated captures are recorded in WARC files. See also Henry Thompson's report and the discussion in the Common Crawl user group.

[x] (protocol-okhttp) reliably annotate content truncated by length limit:
- this happens mostly (100 out of 110) for pages with Content-Encoding: gzip
- no truncation flag is added if the loop to read content chunk by chunk is exited reaching the content limit exactly: verify and open issue to fix this in upstream Nutch (NUTCH-2729)
[x] analyze truncations flagged as "disconnect" due to an IOException
- in 3 analyzed WARC files all records flagged by "disconnect" have either "gzip" content encoding or "chunked" transfer encoding (or even both) - the reason could be also a broken encoding not a "network disconnect". Note: would need also to clarify how to annotate truncations due to protocol-level errors.
- see #13 for more details
[x] always add a Content-Length header to HTTP headers in WARC file, even if there wasn't one in the original HTTP response (eg. for chunked transfer encoding). Implemented in 3663f35.
[x] (in the course) upgrade to latest okhttp library (NUTCH-2728)
[x] port fixes to StormCrawler: DigitalPebble/storm-crawler#757

sebastian-nagel commented 5 years ago

For analysis and verification see

sebastian-nagel commented 5 years ago

Implemented and fixed for August 2019 crawl (CC-MAIN-2019-35). Solution verified on 100 randomly selected WARC files.

tfmorris commented 8 months ago

Implemented and fixed for August 2019 crawl (CC-MAIN-2019-35). Solution verified on 100 randomly selected WARC files.

The above notebook is now here

commoncrawl / nutch