-
Make this a dependency, with its own library.
- [ ] tokenize
- [ ] Tokenizer
- [ ] sanitize
- [ ] Sanitizer
-
I'm using Cohere and unstructured, and I'm receiving that error when trying to load a pdf. It works ok with the simple reader, but not with the options for PDF.
this is the log:
ℹ Received Data …
-
model.tokenize result is different from tranformers tokenizer result
using the same mode Qwen1.5-7B
input_ids1 = model.tokenize(prompt.encode("utf-8"))
input_ids2 = tokenizer([prompt], padding…
-
I have following code:
```python
import dotenv
dotenv.load_dotenv(override=True)
import random
import numpy as np
import torch
import torchvision
from datasets import load_from_disk
…
-
Ref: https://twitter.com/NeoThermic/status/712986560375992320
-
Hello Till,
I have reached out to my IT department; however, they are not able to resolve the below issue. Do you have any suggestions on how to resolve this issue?
Running pythonv 3.9.5, PIP…
-
Related to: https://github.com/huggingface/transformers/issues/25073
In my current project, I'd like to add a special token that doesn't insert a space to the next token.
Currently, I need to spec…
-
Traceback (most recent call last):
File "/home/anaconda3/envs/mgm/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/anaconda3…
-
The actual tokenizer is very fast as it uses a simple regex pattern but at the same time very imprecise.
A better alternative might be to make use of `spaCy`.
Naively tokenize a Pandas Series wi…
-
```
On: 31/08/2017 -2.04
CARD PAYMENT TO SHELL TOTHILL,2.04 GBP, RATE 1.00/GBP ON 29-08-2013
My guess is:
> 6
Traceback (most recent call last):
File "/Users/arturo/Documents/GitHub/BankClas…