search

dominiclovell / boilerpipe

Automatically exported from code.google.com/p/boilerpipe
0 stars 0 forks source link

Page not being parsed correctly <li> the issue. #22

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
Running ArticleExtractor on http://www.seomoz.org/ugc/link-building-management

What is the expected output? What do you see instead?
Expect to see the full article, instead it starts from the last <li> within the 
content of the article, causing a large portion of the article to be stripped.

What version of the product are you using? On what operating system?
Using the appspot version

Please provide any additional information below.
This is not an issue with the default extractor, however the default extractor 
includes comments.

Original issue reported on code.google.com by w...@3whitehats.com on 7 Jun 2011 at 1:50

GoogleCodeExporter commented 9 years ago 
The ArticleExtractor appears to assumes li elements are part of a menu of some 
sort.  Generally this is correct, but it seems we can assume that menus aren 
normally not ordered lists?

Working from that assumption, I was able to modify the code to accept li 
elements that are in an ol into 1 textblock adding the order number before each 
li.  I have not had the chance to test my modifications against a wide variety 
of articles, but it seems to work as expected.

Original comment by tucker...@gmail.com on 15 Mar 2012 at 10:03

GoogleCodeExporter commented 9 years ago 
I think it is relatively safe to assume that, but at the same time I doubt this 
is always the case, some obscure reasons may push a developer to utilise <ol>'s 
instead of <ul>'s.

Also, it is very common for articles to have <ul>'s inside of them - is there 
something that can be added that looks for leading/trailing content blocks 
greater than X length. Or you could alternatively look for lists inside of a 
element which also contain a large volume of text?

Its is unlikely that a div would contain lists as well as a large volume of 
text (where the text is outside of the list element).

Hope that all makes sense?

Original comment by w...@3whitehats.com on 16 Mar 2012 at 7:12

GoogleCodeExporter commented 9 years ago 
Partially fixed in r170. The issue with LIs still needs to be tackled, although 
there are other reasons for this behavior.

Please try again and tell me if you are happy with the results.

Cheers,
Christian

Original comment by ckkohl79 on 21 Mar 2012 at 10:10

GoogleCodeExporter commented 9 years ago 
I tried this using 
http://boilerpipe-web.appspot.com/extract?url=http%3A%2F%2Fwww.seomoz.org%2Fugc%
2Flink-building-management&extractor=ArticleExtractor&output=htmlFragment and 
it doesn't seem to have made much of a difference - has the appspot version 
been updated yet?

Notice how they have used an ol and a ul within the same article.

Original comment by clie...@3whitehats.com on 22 Mar 2012 at 8:48

GoogleCodeExporter commented 9 years ago 
Please try again. It's now live on boilerpipe-web.
(before, it was only on SVN trunk)

Original comment by ckkohl79 on 22 Mar 2012 at 5:48

GoogleCodeExporter commented 9 years ago 
Looking much better, the only issue remaining is it seems to have trimmed out 
two of the <ol> <li>'s - 
http://boilerpipe-web.appspot.com/extract?url=http%3A%2F%2Fwww.seomoz.org%2Fugc%
2Flink-building-management&extractor=ArticleExtractor&output=htmlFragment

These two:
Majestic SEO - Deeper than OSE but contains noisy, unfiltered data.
Official Google Toolbar (PageRank) - Single metric. Infrequently updated.

Great work by the way, works almost perfectly now :-)

Original comment by w...@3whitehats.com on 22 Mar 2012 at 5:53

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
Can something like the ArticleMetadataFilter be used to remove the "82 Thumbs 
Up, 1 Thumbs Down" block?

Original comment by tucker...@gmail.com on 22 Mar 2012 at 8:34

GoogleCodeExporter commented 9 years ago 
Sorry tucker was that directed at me?

Original comment by w...@3whitehats.com on 26 Mar 2012 at 2:59