Research better tokenization/sentence segmentation/adding PoS tagging

[tomschr]

It seems the Natural Language Toolkit (NTLK) could be helpful addition to SDSC.

Some things to watch for:

• good tested Python3 packages (see https://build.opensuse.org/package/show/home:thomas-schraitle/python3-nltk)
• Package dependencies
• Is the corpus good enough?
• Needs some time to get used to it

[ghost]

Also, we should make sure NLTK solves a problem. :)

From my pov, SDSC's language department has room for improvement in these areas:

• Sentence segmentation
• Tokenization

And it would be nice to add a solution for this area:

• Part-of-speech annotation

Finally, we need to check if there are license issues with NLTK (especially if we are using one of their corpora for anything).

[tomschr]

Also, we should make sure NLTK solves a problem. :)

Wouldn't that be the case? ;) I'm still learning NLTK but what I've read so far makes me believe it can be helpful in those areas you've mentioned before.

Especially when regular expressions (which is the main part) has their limits, NLTK could give more accurate results. Of course, it boils down how serious are with style checking and if we want to refactor some parts.

[ghost]

Also, we should make sure NLTK solves a problem. :)

Wouldn't that be the case? ;)

I made this remark mostly because the initial report sounds a little like "let's add NLTK because that sounds fun". Which obviously is not what you wanted to say, hence my mentioning of areas where it might be helpful.

In other words, the initial bug report put the cart in front of the horse. The main focus should be on solving issues, not on adding tools. That is all.

Also, I think it might be helpful to make this a bit more generic. NLTK is probably not the only way to process natural language and it certainly is everything but lightweight. So, if there are alternatives, it would be good to explore those too.

(E.g. what is LanguageTool using?/Can we adapt that without gaining a Java dependency?)

[tomschr]

In other words, the initial bug report put the cart in front of the horse. The main focus should be on solving issues, not on adding tools. That is all.

Right, I agree.

NLTK is probably not the only way to process natural language and it certainly is everything but lightweight. So, if there are alternatives, it would be good to explore those too.

I found Textblob: http://textblob.readthedocs.org / https://github.com/sloria/textblob It looks quite good and seems easier to use than NLTK.

what is LanguageTool using?/Can we adapt that without gaining a Java dependency

Not sure, haven't investigated much. It seems, it's a pure Java library.

[ghost]

• Hm. So, indeed, LanguageTool seems to use its own stack of Java utilities. Not an option.
• Meanwhile, NLTK is Apache 2-licensed. So that is good.
• Even if the corpora are licensed differently, I do not think that would be much of an issue. I am not really sure what we would even use a corpus for. We are not doing research here, we want to consume the results of corpus linguistics. And the training data created from corpora should be using the same license as NLTK itself, Apache 2.
• I found splitta which is apparently rather good for sentence segmentation (of English text). It is bound to be included in NLTK at some later point, though for now, there is only a Python 2 implementation. It consists of a few small scripts and about 5 MB of training data (which already outweighs the size of the current SDSC implementation).

[Vogtinator]

Another alternative might be https://github.com/fnl/segtok It's a python-only library without any dependencies.

[ghost]

More tools to look at:

• Tools with Emacs integration: https://lwn.net/SubscriberLink/692054/de3f720a8e3bf01e/ (thanks, ke)
• Python tool: http://proselint.com/ (In Chromium, use http://proselint.com/write/ for a sample document annotated with ... interesting corrections)
• What Google use: https://github.com/tensorflow/models/tree/master/syntaxnet/syntaxnet/models/parsey_mcparseface

[ghost]

Two more things:

• spaCy: http://spacy.io/ -- currently my front-runner because it appears to be relatively lightweight
• pynini: https://www.oreilly.com/ideas/how-to-get-superior-text-processing-in-python-with-pynini -- probably useful only for a drop-in replacement of regular expressions (afai understood the text)