Speeding up lemmatizer - Githubissues

spacy-pl / utils

Utilities for development polish language support for Spacy.

6 stars 1 forks source link

Speeding up lemmatizer #54

Open MateuszOlko opened 5 years ago

MateuszOlko commented 5 years ago

DOD:

[x] lemmatizer in spacy as default polish lemmatizer
[x] loaded on language load
[x] loading time below 10s
[x] WPS > 4K
[x] add regex expasion to dvc pipeline in lemma sources generation

flayed1 commented 5 years ago

https://github.com/spacy-pl/utils/pull/60 & https://github.com/spacy-pl/spaCy/pull/5 https://github.com/explosion/spaCy/pull/3610 <3

kowaalczyk commented 5 years ago

@Gizzio cannot test this without tagmap merged into the release branch (the rules are POS-specific so the trie search is never executed unless the POS is one of: ['noun', 'verb', 'adj', 'adv', 'part'].