Tokenizer doesn't segment double/triple hyphens correctly

BLLIP / bllip-parser

BLLIP reranking parser (also known as Charniak-Johnson parser, Charniak parser, Brown reranking parser) See http://pypi.python.org/pypi/bllipparser/ for Python module.

http://bllip.cs.brown.edu/

227 stars 53 forks source link

Tokenizer doesn't segment double/triple hyphens correctly #14

Open dmcc opened 10 years ago

dmcc commented 10 years ago

a--b should be a, --, b. Currently it is not split up by the tokenizer.