Closed ysykzheng closed 4 years ago
Fix released as part of v0.13.0. It should sync to Maven central in an hour or two.
Thanks for reporting this. The sample WARC had a couple of other invalid header name variants that I fixed also.
Interesting, recent Common Crawl data is captured using OkHttp and HTTP headers are parsed by okhttp and then serialized again. Obviously, this does not clean up all headers.
Hi, I use this tools to parse CommonCrawl data, but fail.
hit exception:
the data:
related file: crawl-data/CC-MAIN-2020-24/segments/1590347385193.5/warc/CC-MAIN-20200524210325-20200525000325-00000.warc.gz