search

louismullie / treat

Natural language processing framework for Ruby.
Other
1.37k stars 127 forks source link

Tokenize Paragraph #102

Open peddinti opened 9 years ago

peddinti commented 9 years ago

Hi it appears i can only tokenize a sentence but not a paragraph. I have a paragraph and would like to tokenize it, how can i do that? i am also not able to get the sentences in a paragraph to be able to successively tokenize the sentences

louismullie commented 9 years ago

You would need to segment the paragraph first, and then you can tokenize the sentences.

louismullie commented 9 years ago

See the following section from the manual: https://github.com/louismullie/treat/wiki/Manual#segmenters