adbar
commented
4 years ago Hi, thanks for your interest.
According to my benchmark and this external one trafilatura should work better than readability-lxml on average. This can be explained by the way the library is designed:
- It grounds on a cascade of extraction methods: first a series of heuristics for main text discovery and conversion, then two external libraries, then a baseline extraction if nothing worked (basically all text tags such as
<p>). - Since it defaults to
readability-lxmlif extraction fails or appears to have failed, you'll still benefit from the efficiency of this (very good) library,justextis also used as a fallback. In theory (and the benchmarks confirm it)trafilaturahas a better website coverage without sacrificing precision. Its series of heuristics andreadability-lxmlare mostly used, the rest only applies to rare cases. - If you're also interested in extracting comments, this library can do it while
readabilitycannot. - It performs a conversion and can output "homogeneous" (clean) text and tags in XML format, which
readability-lxmldoesn't, sometimes leaving you with "exotic" HTML tags within the content.
I hope it helps!
ydennisy
Hi! Great library, but I am trying to figure out if we should swap from using readability on which this builds. Apart from manually checking the quality for our corpus there is no easy way for me to compare performance. Would you be able to explain the differences between the actual text extraction approaches? Thanks in advance! D