Where to get data2vec 2.0 token dictionary?

[DanielFLevine]

❓ Questions and Help

Before asking:

1. search the issues.
2. search the docs.

What is your question?

Is the token dictionary for the data2vec 2.0 text model available anywhere? The 'data' field in the 'task' dictionary points to '/fsx-wav2vec/abaevski/data/nlp/bookwiki_aml-full-mmap2-bin', and I'm unable to load the checkpoint without the dict.text file. Is the dictionary identical to the RoBERTa 50k BPE or is yours different due to only being trained on BooksCorpus and English Wikipedia? Any help here is appreciated, thanks! @alexeib

Code

model, args, task = fairseq.checkpoint_utils.load_model_ensemble_and_task([path_to_d2v2_text_cp], strict=False)

FileNotFoundError: [Errno 2] No such file or directory: '/fsx-wav2vec/abaevski/data/nlp/bookwiki_aml-full-mmap2-bin/dict.txt'

What have you tried?

Read the README file and tried looking through the checkpoint to find any dictionary.

What's your environment?

• fairseq Version (e.g., 1.0 or main): 0.12.2
• PyTorch Version (e.g., 1.0): 1.13
• OS (e.g., Linux): Ubuntu 20.04.5 LTS
• How you installed fairseq (pip, source): pip install ./ after cloning fairseq repo
• Build command you used (if compiling from source):
• Python version: 3.9.16
• CUDA/cuDNN version: 11.8
• GPU models and configuration: Tesla T4
• Any other relevant information: running in Google Colab

[wnhsu]

Fixed in #5045