Open sebastian-nagel opened 4 years ago
The request records in the CC-NEWS WARC files lack the HTTP protocol version:
GET /path
instead of
GET /path HTTP/1.1
This makes some WARC parsers fail to process the WARC files, see https://groups.google.com/d/msg/common-crawl/hsb90GHq6to/Lv-9-nHAAQAJ.
Fix in Stormcrawler (DigitalPebble/storm-crawler#775) deployed to production, WARC files now contain the HTTP version in the request message.
The request records in the CC-NEWS WARC files lack the HTTP protocol version:
instead of
This makes some WARC parsers fail to process the WARC files, see https://groups.google.com/d/msg/common-crawl/hsb90GHq6to/Lv-9-nHAAQAJ.