-
### Description
When tokenizing a text and decoding these tokens, one can see that tokenization now (as of version 0.14.0) adds one additional starting space to `text` for every call of `Context.Toke…
-
nice work! starred already.
sorry for asking, why replacing the bos_token with empty string?
sample['positive'] = tokenizer.apply_chat_template(
sample['chosen'], tokenize=False, …
-
Could you add a native speech to speech / audio-to-audio support with encoder (tokenizer) and decoder (back to audio waves)
I was able to implement a decoder only model, I first used audio codec to…
-
Hi!
My SSID name contains space, for example "Homer Simpson".
I tried to enter this name through serial monitor, but I get an error in the number of arguments.
```
> said Homer Simpson wifiPassWor…
-
Hi there,
recently I found out that pyenchant can be used alongside pylint to check the spelling in the docstrings. However, I face a small issue. Sometimes in the documentation or in the examples …
-
请问你是怎么解决“def build_vocabulary(spacy_de, spacy_en):
def tokenize_de(text):
return tokenize(text, spacy_de)
def tokenize_en(text):
return tokenize(text, spacy_en)
pr…
-
### What happened?
Note: Discovered by one of the users of Guidance.
If a multibyte UTF-8 character is encoded to two tokens, LlamaCpp is unable to tokenise the byte representation of one of the t…
-
-
Received warning message when normalizing text. Could you pls provide what the message indicates?
**Reproduciple code**:
```python
from nemo_text_processing.text_normalization.normalize import …
-
I am using QuickConverter, and when I use Visiblity for what ever reason I get the following:
`Failed to tokenize expression "!$P ? Visibility.Visible : Visibility.Collapsed". Did you forget a '$'?`
…