Streaming interface to warc files.

[tef]

• Avoid parsing entirety of warc file
• Don't parse http records inside

Any improvements we can make to mean that large and gargantuan warc files can be read and processed speedily

[rajbot]

Many users of the warc library would need to have parsed http headers, so it would be nice to at least have a convenience function to do so. In addition, it might by useful to have a function to stream through the payload and calculate sha1 if the WARC-Payload-Digest header is not present.

I have some changes that implement parsing of http records and calculating sha1 while streaming the payload. However, this happens internal in the library and these changes are not suitable for upstream. https://bitbucket.org/rajbot/warc-tools

[nibrahim]

The warc library at https://github.com/internetarchive/warc has a number of these features.

[tef]

It's GPLed. This is MIT licensed.

Edit: For the record the other major difference is that this library has had to handle more corrupt warcfiles, or weirder variants

• Wrong newlines in records, missing trailing newlines after block
• arcfile with warcrecords.
• gzipped as a whole instead of record by record.

(and the http library handles far too much weirdness)

That said, the interface to warc is /far/ nicer.