-
here is the error log.
running with exmple workflow
-
`Traceback (most recent call last):
File "/home/rohanp/MHCRoBERTa/process_MHC_pep.py", line 114, in
spm.SentencePieceTrainer.Train('--input=uniprot_pretraining_data.txt --model_prefix=m_revie…
-
Initializing Chat
Loading VIT
Loading VIT Done
Loading Q-Former
Loading Q-Former Done
Loading LLAMA
miniconda3/envs/fit/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:1714:…
-
I was able to successfully compile the library but when I use it as indicated in the example folder I am having the following errors:
ld: error: undefined symbol: sentencepiece::SentencePieceProces…
-
## 🐛 Bug
### To Reproduce
My Python environment is 3.12.4 and sentencepiece dependency require python < 3.12 and there is a new sentencepiece 0.2.0 released when can we expect new release o…
-
# ❓ Questions & Help
HI, I can not save model and vocab when using spm.SentencePieceTrainer.Train.
## Details
This is my config:
python3 ./openspeech_cli/hydra_train.py dataset=librispeech…
-
Amazing work! Currently the spmtrain in build_tokenizer doesn't work, cuz I think it needs a local installation of sentencepiece to be able to use the command. Is there a specific reason you chose Goo…
-
## Issue Description
When attempting to convert a slow tokenizer to a fast tokenizer using the `transformers` library, an error occurred due to the inability to convert from Tiktoken. The error sug…
-
hello
I want to build a speech translation model for an Arabic dialect using the fairseq toolkit, so I start by preparing my data to start to train my model. For preparing data for training I have in…
-
I recently encountered some compatibility issue when using `sentencepiece v0.2.0` together with latest `transformers` and `tensorflow` packages. When I ran some Python script that imports `AutoProcess…