search

simongray / StatementAnnotator

Custom annotator for Stanford CoreNLP that annotates sentences with the underlying statements contained within them.
4 stars 0 forks source link

Tokenizer separates forward slashed paths into separate tokens #18

Open simongray opened 8 years ago

simongray commented 8 years ago

example: "/r/videos" becomes "/" and "r/videos" when tokenized.

Figure out how to undo this.

Resource: http://nlp.stanford.edu/software/tokenizer.shtml