Allow to access WARC filename, record offset and length

commoncrawl / cc-pyspark

Process Common Crawl data with Python and Spark

MIT License

406 stars 86 forks source link

Closed sebastian-nagel closed 5 years ago

sebastian-nagel commented 5 years ago

Allow to access WARC filename and from ArchiveIterator record offset and length (see #6)

introduce customizable method iterate_records(warc_file_uri, archive_iterator) which iterates over WARC record and calls process_record(record)
document pitfall: accessing offset and length must be done after WARC record is processed