Handle base href - Githubissues

openzim / warc2zim

Command line tool to convert a file in the WARC format to a file in the ZIM format

https://pypi.org/project/warc2zim/

GNU General Public License v3.0

44 stars 4 forks source link

Handle base href #243

Closed benoit74 closed 5 months ago

benoit74 commented 5 months ago

Fix #238

Changes:

detect base href in HTML pages head
use it to properly rewrite URLs found in the HTML page
- includes multiple changes to properly pass an ArticleUrlRewriter to all rewriters and get rid of the UrlRewriterProto in most situations
rewrite the base to remove the href (for simplicity) but keep the target

Remark:

detection in done with Beautifulsoup to more easily handle all edge cases around multiple base tags in a single HTML document, ...
- Beautifulsoup is limited to 4.9.3 due to scraperlib limiting it to < 4.10

benoit74 commented 5 months ago

why do we need the SimpleUrlRewriter?

This is kinda a mock for tests ; it was previously done with a lambda which was just returning the same url as passed, but it was kinda a hack and this does not work very nicely now that we need to pass both url and base_href to the rewriter.