'DistilBertTokenizer' object has no attribute 'max_len'

[mymusise]

Got an error when running the example:

from moverscore_v2 import plot_example
reference = 'they are now equipped with air conditioning and new toilets.'
translation = 'they have air conditioning and new toilets.'
plot_example(True, reference, translation)

I think maybe the API has changed, model_max_length now.

[mymusise]

fix in https://github.com/AIPHES/emnlp19-moverscore/pull/18

[sarwart]

I am currently using this on Google colab. Can you please be more specific how to solve this on colab? I am using the latest transformers. I got an error when I tried the mentioned example in the following line

"idf_dict_hyp = get_idf_dict(translations)"

Also, how can I specify a different mode like ClinicalBERT to compute the moverscore?

Tabinda

[mymusise]

@sarwart try my branch: !pip uninstall moverscore !pip install https://github.com/mymusise/emnlp19-moverscore/archive/master.zip

To use a different mode like ClinicalBERT, just define an env like this:

import os 
os.environ['MOVERSCORE_MODEL'] = "emilyalsentzer/Bio_ClinicalBERT"
# or 
# os.environ['MOVERSCORE_MODEL'] = "albert-base-v2"
# and so on
from moverscore_v2 import get_idf_dict
idf_dict_hyp = get_idf_dict(translations)

[andyweizhao]

@mymusise, thanks a lot! The PR is merged. Cheers :-)