search

dominiclovell / boilerpipe

Automatically exported from code.google.com/p/boilerpipe
0 stars 0 forks source link

Library does not produce same results as http://boilerpipe-web.appspot.com/ #46

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
1) Go to http://boilerpipe-web.appspot.com/
2) Type in http://arstechnica.com/ as the URL.
3) Use article extractor and HTML (extract fragment)
4) See a nice list of articles on that page

Compare to:
1) Download latest boilerpipe svn.
2) Use the following code:

        final URL url = new URL("http://arstechnica.com/");
        final ArticleExtractor articleExtractor = ArticleExtractor.INSTANCE;
        final HTMLHighlighter htmlHighlighter = HTMLHighlighter.newExtractingInstance();
        final String xhtml = htmlHighlighter.process(url, articleExtractor);
3) xhtml only contains 1 article.

Are there settings that need to be changed? Or is there a code update that 
hasn't been checked in?

Original issue reported on code.google.com by paul.jak...@gmail.com on 30 Mar 2012 at 2:50

GoogleCodeExporter commented 9 years ago 
It looks like the issue is the KeepLargestBlockFilter which rejects every block 
except the largest. While taking out this filter in the library should return 
results closer to http://boilerpipe-web.appspot.com/, it looks like that will 
cause some non-article things that are rejected by 
http://boilerpipe-web.appspot.com/ to make it through.

Is there anyplace I can go to see what filters are being by the article 
extractor used by http://boilerpipe-web.appspot.com/?

Original comment by paul.jak...@gmail.com on 30 Mar 2012 at 4:46

GoogleCodeExporter commented 9 years ago 
I'm also unable to get the same results using the HTMLHighlighter in extraction 
mode.  The web API (http://boilerpipe-web.appspot.com) clearly states that:

"This Web Application probably uses a more recent version than the released 
versions in the Boilerpipe Google Code page. You might thus get slightly 
different (hopefully better) results."

In the web demo, are you using the HTMLHighligher to extract HTML, or have you 
added a different approach to boilerpipe-core?  If changes have been made to 
core, I'd be happy to contribute to the open source version.  Hints would be 
appreciated =)

Original comment by matth...@linksmart.com on 19 Oct 2012 at 5:49

GoogleCodeExporter commented 9 years ago 
I got the similar issue.
When trying the URL "http://www.hokkaido-np.co.jp/news/donai/424760.html"
With ArticleExtractor and "Plain Text" output

Library code did not produce same results as http://boilerpipe-web.appspot.com/

Original comment by jc...@geosemble.com on 6 Dec 2012 at 12:02

GoogleCodeExporter commented 9 years ago 
Building from source will fix this issue.

Original comment by pettib...@gmail.com on 24 Jan 2013 at 8:33

GoogleCodeExporter commented 9 years ago 
I have done a build from SVN. Still Japanese URL gives bad results. Same works 
gr8 on the Web App. Code snippet below:
    URL url = new URL("http://d.hatena.ne.jp/mkusunok/20130817/p1");
    String text = ArticleExtractor.INSTANCE.getText(url);

Original comment by navra...@gmail.com on 7 Oct 2013 at 7:24