internetarchive / warcprox

WARC writing MITM HTTP/S proxy
380 stars 54 forks source link

Skip CDX dedup check for captures with empty payload #79

Closed vbanos closed 6 years ago

vbanos commented 6 years ago

Check for not empty captured content (payload_size() > 0) before creating a new thread and running a CDX dedup request.

Most dedup modules perform the same check to avoid unnecessary dedup requests.