No more failure due to url rewriting issue

openzim / warc2zim

Command line tool to convert a file in the WARC format to a file in the ZIM format

https://pypi.org/project/warc2zim/

GNU General Public License v3.0

44 stars 4 forks source link

Closed benoit74 closed 4 months ago

benoit74 commented 4 months ago

Fix #247

all URL rewriting issues are simply logged, but they do not cause the scraper to fail
no more special handling of hostname rewriting issues
- try to decode hostname only if it starts with "xn--" (the indicator used for IDNA puny-encoded see here)

benoit74 commented 4 months ago

Merging right away as this has been discussed live with rgaudin