Open anjackson opened 2 years ago
This came up recently in IIPC Slack when trying to diagnose why warcheology was reporting a corrupted WARC file, and warcio was not. It appeared that the WARC file was truncated as a result of a browsertrix-crawler container exiting abnormally, and not closing the GZIP file properly...
In case it's helpful to have a test script (which doesn't emit a warning that I can see):
from warcio.archiveiterator import ArchiveIterator
with open('test.warc.gz', 'rb') as stream:
for i, record in enumerate(ArchiveIterator(stream)):
print(i, record.rec_headers.get_header('WARC-Target-URI'))
if record.rec_type == 'response':
content = record.content_stream().read()
And here's a test file: test.warc.gz
gunzip
on the other hand does notice:
$ gunzip --test test.warc.gz
gunzip: truncated input
gunzip: test.warc.gz: uncompress failed
Wow, I'd totally forgotten about this!
Seems like there's a hook in the underlying Python library to spot this case:: https://docs.python.org/3/library/zlib.html#zlib.Decompress.eof
Decompress.eof A boolean indicating whether the end of the compressed data stream has been reached. This makes it possible to distinguish between a properly formed compressed stream, and an incomplete or truncated one. New in version 3.3.
But it's not clear to me how to weave that in here...
@edsu what record in test.warc.gz is the truncated one? And where can I find warcheology? Thanks.
I believe it's the last record. If you try to gunzip the file, you should see the error error right at the end?
I'm not really familiar with it but here is the warchaeology repo: https://github.com/nlnwa/warchaeology
@edsu thanks for adding a simple test and @anjackson for looking up the .eof
property!
With that, I think detecting this case can be done as follows:
diff --git a/warcio/archiveiterator.py b/warcio/archiveiterator.py
index 484b7f0..451f182 100644
--- a/warcio/archiveiterator.py
+++ b/warcio/archiveiterator.py
@@ -113,7 +113,13 @@ class ArchiveIterator(six.Iterator):
yield self.record
- except EOFError:
+ except EOFError as e:
+ if self.reader.decompressor:
+ if not self.reader.decompressor.eof:
+ sys.stderr.write("warning: final record appears to be truncated")
+
empty_record = True
self.read_to_end()
But, what is the desired behavior be more generally?
warcio check
, seems like it should return an errorgunzip
behavior is definitely not desirable, as that fails to unzip any record even if only last one is invalid.It sort of depends on how the WARC is being used:
One of the most likely problems we see is failed transfers leading to truncated WARC.GZ files. We can spot this with
gunzip -t
but it would be good ifwarcio check
also raised this as a validation error. My tests so far have indicated that the warcio and cdxj-indexer etc. tools all skip over these errors silently.