search

stefan-it / german-gpt2

German GPT-2 model
MIT License
32 stars 4 forks source link

No Vocab #13

Open SchahinRohani opened 2 years ago

SchahinRohani commented 2 years ago

Hello, i am trying to use the model in julia/transformers.jl lib from the HuggingFace dbmdz/german-gpt2 After getting a error when loading the model, i found that the main branch doesnt have a vocab file.

Shouldnt it have vocab file, or should i add a vocab file?

stefan-it commented 1 year ago

Hi @Schahin-R ,

the vocab is stored in the tokenizer.json file that can be found here:

https://huggingface.co/dbmdz/german-gpt2/tree/main

(The model is using a Fast Tokenizer, so merges.txt and vocab.json are not available compared to the original GPT-2)