search

internetarchive / warcprox

WARC writing MITM HTTP/S proxy
378 stars 54 forks source link

Warc close api #109

Closed nlevitt closed 5 years ago

nlevitt commented 5 years ago 
remove --writer-threads option

Support for multiple writer threads was broken, and profiling had shown
it to be of dubious utility.
https://github.com/internetarchive/warcprox/issues/101
https://github.com/internetarchive/warcprox/wiki/benchmarking-number-of-threads

and

WarcWriterProcessor.close_for_prefix()

New API to allow some code from outside of warcprox proper (in a
third-party plugin for example) can close open warcs promptly when it
knows they are finished.