-
```
When writing a tokenization unit test for the ClearTK wrappers for ClearNLP, I
found an inconsistency between OpenNLP's tokenization and ClearNLP's.
Consider the string:
String s = "\"John & Mar…
-
```
When writing a tokenization unit test for the ClearTK wrappers for ClearNLP, I
found an inconsistency between OpenNLP's tokenization and ClearNLP's.
Consider the string:
String s = "\"John & Mar…
-
An ongoing discussion of tokenization and its relation to morphlogy is happening in issue #42 It really should be happening here.
linas updated
6 years ago
-
Hello everyone, I want to implement a Fast WordPiece Tokenization algorithm introduced by Google.
# Fast WordPiece algortihm
Google introduced a new algorithm called LinMaxMatch for WordPiece tok…
-
As stated in the VSCode [Semantic Highlight Guide](https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide):
>Semantic tokenization allows language servers to provide additio…
-
### What happened?
The llama.cpp tokenizer for Phi-3 has odd behavior, where re-tokenizing the same text over and over keeps adding whitespaces to the first non-BOS token. This has several issues:
…
-
I am getting maximum recursion depth error after running this following command:
python qlora.py --model_name_or_path decapoda-research/llama-7b-hf
And this is the error I got:
File "/home/at…
-
test_tokenization tests fail with the following error:
```
if tensor_type == TensorType.TENSORFLOW:
if not is_tf_available():
raise ImportError(
> …
-
Which methods does aitextgen support for tokenization (BPE, wordpiece etc..)?
If only one, how can we expand to use others?
Thanks!
-
```
When writing a tokenization unit test for the ClearTK wrappers for ClearNLP, I
found an inconsistency between OpenNLP's tokenization and ClearNLP's.
Consider the string:
String s = "\"John & Mar…