search

karussell / snacktory

Readability clone in Java
461 stars 159 forks source link

Provide optional extraction directives #24

Open bejean opened 11 years ago

bejean commented 11 years ago

What about provide optional extraction directives ?

In a majority of cases the extraction algorithm woks great. But for some web sites it can fail to extract relevant content. For these web sites it could be possible to "help" snacktory to focus on a specific part of the page content by providing it a Jsoup selector. For instance, we could have something like :

ArticleTextExtractor extractor = new ArticleTextExtractor(); extractor.setTextSelector("div.article_content"); extractor.setTitleSelector("h2", "first"); String dateRegEx = "xxxx"; extractor.setDateSelector("#published", dateRegEx);

JResult res = extractor.extractContent(rawData); text = res.getText(); title = res.getTitle(); date = res.getDate();

karussell commented 11 years ago

Hmmh I don't find this solution that useful as one could simply use jsoup directly for those failing sites. Also I would rather adapt the core to include the failing site. Let me think about it.

bejean commented 11 years ago

Provide a scope to snacktory for the text extraction means to use the snacktory algorithm within this scope. We still need snacktory algorithm.

karussell commented 11 years ago

I see what you mean!