Open puigru opened 2 years ago
Yes, unfortunately, the wget CDX format has never been supported. The cdx-convert was designed to support the cdx 2015 format you've referenced, and older formats, but not the wget format currently. I'll revise this as a request to support that format.
Describe the bug
Running
wb-manager cdx-convert
on CDX indexes created by Wget 1.20.3 creates broken CDXJ indexes.Steps to reproduce the bug
wget --mirror --warc-file=test --warc-cdx http://motherfuckingwebsite.com/
wb-manager cdx-convert .
Expected behavior
The CDXJ file would not be mangled
Environment
Additional context
Example CDX file created by Wget:
CDXJ file created by wb-manager:
Notice how the fields have been shifted, "length" contains the "compressed arc file offset", "offset" contains "file name", etc. See: https://iipc.github.io/warc-specifications/specifications/cdx-format/cdx-2015/