search

cisnlp / simalign

Obtain Word Alignments using Pretrained Language Models (e.g., mBERT)
MIT License
345 stars 47 forks source link

Move to transformers 3 #12

Closed masoudjs closed 3 years ago

masoudjs commented 3 years ago

Changed the code to use the new transformers 3.1. The embedding loader can now tokenize and generate embeddings in batches.

masoudjs commented 3 years ago

@pdufter I have updated the code.