Title detection: Treat non-breaking space as whitespace

[GoogleCodeExporter]

Character.isWhitespace(String) does not consider the non breaking space 
character (160) to be whitespace.  This causes whitespace to not be correctly 
trimmed when the non breaking space character is involved.  This can cause 
DocumentTitleMatchClassifier to miss a title match as well as other whitespace 
related problems.

The following article uses the character in the title and a few other places:
http://espn.go.com/dallas/nfl/story/_/id/7560381/do-anthony-spencer

Original issue reported on code.google.com by tucker...@gmail.com on 20 Mar 2012 at 2:58

Attachments:

• BoilerpipeHTMLContentHandler.java.patch

[GoogleCodeExporter]

Hi tuckerpmt,

thanks for reporting this. Especially the link to the article helped me 
figuring out what the problem was.
I have fixed this in r167, although in a slightly different way.

Cheers,
Christian

Original comment by ckkohl79 on 21 Mar 2012 at 9:10

• Changed title: Title detection: Treat non-breaking space as whitespace
• Changed state: Fixed
• Added labels: Type-Enhancement
• Removed labels: Type-Patch

[GoogleCodeExporter]

Issue 38 has been merged into this issue.

Original comment by ckkohl79 on 21 Mar 2012 at 9:10

[GoogleCodeExporter]

Issue 39 has been merged into this issue.

Original comment by ckkohl79 on 21 Mar 2012 at 9:10

[GoogleCodeExporter]

Issue 42 has been merged into this issue.

Original comment by ckkohl79 on 21 Mar 2012 at 9:11

[GoogleCodeExporter]

Issue 40 has been merged into this issue.

Original comment by ckkohl79 on 21 Mar 2012 at 9:11

[GoogleCodeExporter]

Glad to help! :)

Though I am curious as to why you chose to scan all of the characters again 
instead of handle it where you currently are converting other whitespace 
characters to the standard space character?

Original comment by tucker...@gmail.com on 21 Mar 2012 at 10:44