search

adbar / trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
https://trafilatura.readthedocs.io
Apache License 2.0
3.67k stars 262 forks source link

Fix XPath expression in subtree #289

Open adbar opened 1 year ago

adbar commented 1 year ago

In core.py the XPath expression is unnecessarily wide:

ptest = subtree.xpath(//p//text())

In this part, it is marked as //p//text() so It's looking for the root's p-tag rather than the subtree's p-tag.

.//p//text() would be the right approach.

Originally posted by @jake-yi in https://github.com/adbar/trafilatura/discussions/287

adbar commented 1 year ago

It changes the extraction parameters so the thresholds below have to be revised.