search

hsiehsh168168 / warc-tools

Automatically exported from code.google.com/p/warc-tools
0 stars 0 forks source link

SRS 48 — A set of command line tools incorporating libwarc shall perform migration of "curl" archives to WARC-records. #54

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
SRS 48 — A set of command line tools incorporating libwarc shall perform
migration of "curl" archives to WARC-records.

Original issue reported on code.google.com by gordon.p...@gmail.com on 27 Jul 2008 at 10:13

GoogleCodeExporter commented 8 years ago 
Curl only downloads one file at a time, so little need for integration with 
libwarc.

Instead, Hanzo has created a python command line tool (url2warc.py) for 
downloading
multiple files using curl (using any of the many protocols available to 
libCurl) and
storing in a WARC file.

Original comment by gordon.p...@gmail.com on 24 Oct 2008 at 12:39