-
The goal is to discretize the speech data from the French Librispeech dataset you previously worked on (@abheesht17 & Adithiya): https://huggingface.co/datasets/abheesht/librispeech_fr. Remember that…
-
Consider the following code:
```
from friendly_traceback import token_utils as t
toks = t.tokenize('b = 0o1876')
for tok in toks:
print(repr(tok))
```
With Python 3.10 we get
```
type=1 (…
-
write functions use AutoTokenizer to process texts with models from hugging face.
-
Thanks for the fantastic work. When I tried to learn your work by re-implementing the code, finding that the command
```
python tools/preprocessing/a9_tokenize.py --root-path ./data/tumtraf-i-no-sp…
-
```python3
from sacremoses import MosesTokenizer
print(MosesTokenizer(lang='en').penn_tokenize("-LRB- This is very nice -RRB-"))
```
I got the following error. And I found changing `lang='en'`…
-
Traceback (most recent call last):
File "run.py", line 28, in
train_data, dev_data, test_data = build_dataset(config)
File "D:\PYTHON_PROGRAMME\Bert-Chinese-Text-Classification\utils.py", …
-
### Product
SnapPay
### Page
API Explorer
### Description
From a customer:
"I have added a Merchant for sandbox env. Now when I am trying to run tokenize
api on explorer and I am getting 401 …
-
I think the tokenizer is no longer supported in the pipeline , please help me with this issues , I directly took the code from https://huggingface.co/UCSC-VLAA/HQ-Edit
-
1. Seems like there's a redundant code here.
![image](https://github.com/xlang-ai/instructor-embedding/assets/91165212/6406f92d-3a5f-42d2-b6cc-b1b0cd5355b3)
2. Tokenize method not working well for c…
-
Message: 'Error occurs in lazy tokenize:' 这个报错怎么解决