Closed danieldk closed 4 years ago
Before merging, it would be nice if there was a release of edit_tree on crates.io, so that we do not have to rely on a git version. In
edit_tree
, the nameTreeNode
is very generic. Maybe it would be clearer to rename this type toEditTree
? (Each subtree is also an edit tree of some sort.)
Sure.
Decoding consists of (attempting to) apply the edit tree to a form.
My first impression was that falling back to the form instead of falling back to other edit-trees works better, didn't do exhaustive experiments though.
When words are to long (longer than 40 characters by default), any character beyond the maximum length is cut off. For lemmatization, it would probably be nicer to do something more sophisticated (in a separate PR).
In my rust seq2seq lemmatizer, I gave the form as lemma for words above a certain length, most of these words (IIRC mostly in lassy) were links or other non-inflected things. In other, not that extreme, cases giving an option to cut the middle of the word may be an option.
My first impression was that falling back to the form instead of falling back to other edit-trees works better, didn't do exhaustive experiments though.
That sounds reasonable! Now nothing is done, which is probably the worst strategy (no lemma). I'll update the PR and make it configurable to do one of:
In my rust seq2seq lemmatizer, I gave the form as lemma for words above a certain length, most of these words (IIRC mostly in lassy) were links or other non-inflected things. In other, not that extreme, cases giving an option to cut the middle of the word may be an option.
Both seem reasonable!
Before merging, it would be nice if there was a release of edit_tree on crates.io, so that we do not have to rely on a git version.
This encoder encodes lemmas as edit trees that can be applied to a form (using Tobias Pütz' edit_tree crate).
Decoding consists of (attempting to) apply the edit tree to a form.
This turned out to be easy to do. Random comments:
edit_tree
oncrates.io
, so that we do not have to rely on a git version.edit_tree
, the nameTreeNode
is very generic. Maybe it would be clearer to rename this type toEditTree
? (Each subtree is also an edit tree of some sort.)