The issue is self-explanatory, this behavior is a bit unexpected.
I am not sure if it's a bug or a design decision.
It happens in _lemmatize_sent as well.
def _lemmatize_doc(self, in_doc, obmit_tag=False): # assuming input is a document
if type(in_doc) == str: # in_doc is a raw string in this case
in_doc = self._tokenize_doc(in_doc)
in_doc = self._posdep_doc(in_doc)
lemmatized_doc = self._lemma_model[self._config.active_lang].predict(in_doc, obmit_tag)
gc.collect()
return lemmatized_doc
Possible fix (perhaps with some condition in cases where the data is already tagged):
def _lemmatize_doc(self, in_doc, obmit_tag=False): # assuming input is a document
if type(in_doc) == str: # in_doc is a raw string in this case
in_doc = self._tokenize_doc(in_doc)
in_doc = self._posdep_doc(in_doc)
lemmatized_doc = self._lemma_model[self._config.active_lang].predict(in_doc, obmit_tag)
gc.collect()
return lemmatized_doc
https://github.com/nlp-uoregon/trankit/blob/f6b916b39decb3bc8b87b0c4bf100f3de41e3d24/trankit/pipeline.py#L980
The issue is self-explanatory, this behavior is a bit unexpected. I am not sure if it's a bug or a design decision. It happens in _lemmatize_sent as well.
Possible fix (perhaps with some condition in cases where the data is already tagged):