Closed GoogleCodeExporter closed 8 years ago
The ArticleExtractor appears to assumes li elements are part of a menu of some
sort. Generally this is correct, but it seems we can assume that menus aren
normally not ordered lists?
Working from that assumption, I was able to modify the code to accept li
elements that are in an ol into 1 textblock adding the order number before each
li. I have not had the chance to test my modifications against a wide variety
of articles, but it seems to work as expected.
Original comment by tucker...@gmail.com
on 15 Mar 2012 at 10:03
I think it is relatively safe to assume that, but at the same time I doubt this
is always the case, some obscure reasons may push a developer to utilise <ol>'s
instead of <ul>'s.
Also, it is very common for articles to have <ul>'s inside of them - is there
something that can be added that looks for leading/trailing content blocks
greater than X length. Or you could alternatively look for lists inside of a
element which also contain a large volume of text?
Its is unlikely that a div would contain lists as well as a large volume of
text (where the text is outside of the list element).
Hope that all makes sense?
Original comment by w...@3whitehats.com
on 16 Mar 2012 at 7:12
Partially fixed in r170. The issue with LIs still needs to be tackled, although
there are other reasons for this behavior.
Please try again and tell me if you are happy with the results.
Cheers,
Christian
Original comment by ckkohl79
on 21 Mar 2012 at 10:10
I tried this using
http://boilerpipe-web.appspot.com/extract?url=http%3A%2F%2Fwww.seomoz.org%2Fugc%
2Flink-building-management&extractor=ArticleExtractor&output=htmlFragment and
it doesn't seem to have made much of a difference - has the appspot version
been updated yet?
Notice how they have used an ol and a ul within the same article.
Original comment by clie...@3whitehats.com
on 22 Mar 2012 at 8:48
Please try again. It's now live on boilerpipe-web.
(before, it was only on SVN trunk)
Original comment by ckkohl79
on 22 Mar 2012 at 5:48
Looking much better, the only issue remaining is it seems to have trimmed out
two of the <ol> <li>'s -
http://boilerpipe-web.appspot.com/extract?url=http%3A%2F%2Fwww.seomoz.org%2Fugc%
2Flink-building-management&extractor=ArticleExtractor&output=htmlFragment
These two:
Majestic SEO - Deeper than OSE but contains noisy, unfiltered data.
Official Google Toolbar (PageRank) - Single metric. Infrequently updated.
Great work by the way, works almost perfectly now :-)
Original comment by w...@3whitehats.com
on 22 Mar 2012 at 5:53
[deleted comment]
Can something like the ArticleMetadataFilter be used to remove the "82 Thumbs
Up, 1 Thumbs Down" block?
Original comment by tucker...@gmail.com
on 22 Mar 2012 at 8:34
Sorry tucker was that directed at me?
Original comment by w...@3whitehats.com
on 26 Mar 2012 at 2:59
Original issue reported on code.google.com by
w...@3whitehats.com
on 7 Jun 2011 at 1:50