Create and test Hadoop 020/3 CDX indexer

ukwa / ukwa-manage

Shepherding our web archives from crawl to access.

Apache License 2.0

10 stars 5 forks source link

Closed anjackson closed 2 years ago

anjackson commented 2 years ago

Create a PyWB-based CDX indexer command-line tools that:

[x] Runs on Hadoop 020 and Hadoop 3.
[x] Records indexing status in TrackDB.
[x] Accurately generates both the offset and compressed record length.
[x] Extracts some basic info from the WARCs, to add to TrackDB.
[x] ~~Verify indexing e.g. checking first and last records are present and accurate in CDX index.~~

anjackson commented 2 years ago

Shifting verification to a second phase.