search

janih / boilerpipe

Boilerplate Removal and Fulltext Extraction from HTML pages
2 stars 0 forks source link

Ignore FORM tags in HTMLHighlighter #44

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
When using HTMLHighlighter some times boilerpipe keeps some artifacts related 
coming from FORM and LABEL tags.

This can be easily prevented by addding a new ignorable element to TAG_ACTIONS 
map in HTMLHighlighter.java:

TAG_ACTIONS.put("FORM", TA_IGNORABLE_ELEMENT);

Original issue reported on code.google.com by xavi.beu...@gmail.com on 24 Mar 2012 at 6:40

GoogleCodeExporter commented 9 years ago 
Issue 45 has been merged into this issue.

Original comment by ckkohl79 on 25 Mar 2012 at 2:12

GoogleCodeExporter commented 9 years ago 
Adding FORM as an ignorable element at the highlighter (but not at the 
Extractor itself) has two disadvantages:

1. The highlighted HTML will not be consistent with the TextDocument's content 
information (i.e., it should rather go into DefaultTagActionMap)
2. FORM may span over many otherwise relevant text blocks. Adding FORM as 
TA_IGNORABLE_ELEMENT significantly reduces extraction accuracy (with L3S-GN1, 
avg. token-level F1 goes down by about 5% to 7%).

Could you please give examples (URLs) where boilerpipe fails currently?
Please try the very latest version from trunk, or -- preferably -- the version 
at http://boilerpipe-web.appspot.com/

Original comment by ckkohl79 on 25 Mar 2012 at 2:19

GoogleCodeExporter commented 9 years ago 
Adding FORM as an ignorable element at the highlighter (but not at the 
Extractor itself) has two disadvantages:

1. The highlighted HTML will not be consistent with the TextDocument's content 
information (i.e., it should rather go into DefaultTagActionMap)
2. FORM may span over many otherwise relevant text blocks. Adding FORM as 
TA_IGNORABLE_ELEMENT significantly reduces extraction accuracy (with L3S-GN1, 
avg. token-level F1 goes down by about 5% to 7%).

Could you please give examples (URLs) where boilerpipe fails currently?
Please try the very latest version from trunk, or -- preferably -- the version 
at http://boilerpipe-web.appspot.com/

Original comment by ckkohl79 on 25 Mar 2012 at 2:19