Internationalization and ngram performance

dbpedia-spotlight / pignlproc

Apache Pig utilities to build training corpora for machine learning / NLP out of public Wikipedia and DBpedia dumps.

17 stars 14 forks source link

Closed jodaiber closed 11 years ago

jodaiber commented 11 years ago

This pull request has three parts:

renamed and moved pig scripts + params files to be clearer for new users
Added all Redirect patterns from DEF to AnnotatingMarkupParser.java
Added RestrictedNGramGenerator, which only produces ngrams that are know surface forms (reduces disk usage and increases speed)

maxjakob commented 11 years ago

Additionally moved *.params to examples/indexing, so they reside next to their respective scripts.

Thanks Jo! :+1: