search

nlp-uoregon / trankit

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
Apache License 2.0
724 stars 99 forks source link

How to load my custom pretrained dependency parser trained using Trankit for inference? #55

Closed Jivnesh closed 2 years ago

Jivnesh commented 2 years ago

Hi @minhhdvn @laiviet @b-mcdowell

I am planning to deploy my pretrained dependency model (trained using Trankit). Can you please help with this? How to load my custom pretrained dependency parser trained using trankit for the inference? I am only interested in this task. Not the complete pipeline?

Jivnesh commented 2 years ago

Hi @minhhdvn @laiviet @b-mcdowell

import trankit
from trankit import Pipeline
trankit.verify_customized_pipeline(
    category='customized-mwt', # pipeline category
    save_dir='./save_dir/san', # directory used for saving models in previous steps
    embedding_name='xlm-roberta-base' # embedding version that we use for training our customized pipeline, by default, it is `xlm-roberta-base`
)
p = Pipeline(lang='customized-mwt', cache_dir='./save_dir/san')
fr_output = p.posdep(['सीता लक्ष्मणः च तान् तान्नमतः'.split()])
fr_output['sentences'][0]['tokens']

I tried this to load the pretrained model using their script to load pipeline. I have not added any extra snippet of code to save anything. I have simply commented the snippets of code related to other task which were giving the errors. I think its working as expected.