search

vncorenlp / VnCoreNLP

A Vietnamese natural language processing toolkit (NAACL 2018)
Other
587 stars 145 forks source link

word segmentation without sentence segmentation? #17

Closed trangtv57 closed 5 years ago

trangtv57 commented 5 years ago

I want just word segmentation my data, not split sentence by "." or other punctuation. But I don't know what's option for this. Or do you need sentence segmentation for word segmentation better ? Thanks you

tienthanhdhcn commented 5 years ago

Yes, it is. The process is to segment the tokenized text into sentences first and then segment the sentences into words. To meet your need, you can write some simple snippet in Java or Python to process the output of the model. Cheers, T