search

chengchingwen / Transformers.jl

Julia Implementation of Transformer models
MIT License
526 stars 75 forks source link

Tokenizer not support yet | Example GPT2_TextGeneration #112

Closed SchahinRohani closed 2 years ago

SchahinRohani commented 2 years ago

Hello, tried to run the new GPT2_TextGeneration Example and getting an error:

ERROR: LoadError: tokenizer not support yet. Stacktrace: [1] error(s::String) @ Base ./error.jl:33 [2] load_hgf_pretrained(name::String) @ Transformers.HuggingFace ~/.julia/packages/Transformers/jtjKq/src/huggingface/HuggingFace.jl:58 [3] top-level scope @ ~/Projects/Julia/Transformers/example/GPT2_TextGeneration/text_generation.jl:10

Seems like the new implemented gpt2 tokenizer doesnt get recognized.

How to get the Example working?

chengchingwen commented 2 years ago

What version of Transformers.jl are you using?

SchahinRohani commented 2 years ago

Just checked the Version and updated some dependencies, but still not working.

(@v1.7) pkg> status Transformers Status ~/.julia/environments/v1.7/Project.toml [21ca0261] Transformers v0.1.16

chengchingwen commented 2 years ago

You seems to be using v0.1.16 and would need v.0.1.21 to get that work. Try pkg> add Transformers@v0.1.21 and see if anything is blocking the update.

SchahinRohani commented 2 years ago

Solved, thank you for the fast response!