This push addresses a serious difference in the tokenization, the one we used was the sentencepiece llama tokenizer. It is not able to tokenizer for Falcon, almost any long word was broken up (including special words) and international content was almost always complete garbage afterward.
This is a serious issue to model quality, so it had to go.
This is a large change, I've spent 20+ hours first reversing what BPE in GPT-2 regex mode really does, then reimplementing it in C++ without the use of regular expressions and unicode libraries. Could have used regexp but it felt wrong to add such a chunky slowness into it.
The whole task felt like reinventing the wheel while getting punched constantly.
So I made a slim unicode library to work with and identify unicode data, bpe-decoders and -encoders similar to the special GPT2-BPE ones, a lookahead parser that should perform identical to the GPT/Falcon regular expression: 's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+.
In addition I created a new magic and file version for Falcon ggllm.cpp called "GGCC" V0, there is no way around that now as the falcon tokenizer requires 65,000 merging sub-token pairs to operate.
The model loader supports two modes:
1) old compatibility mode (also for quantizing/converting to GGCC) - you need to have the original tokenizer.json file in the same directory as the model binary. The original falcon-7b tokenizer.json should work for all finetunes because they do not change bpe merging. I did not integrate it into the python code, it's still creating llama ggml V1 to be converted.
The json parser I wrote is absolute minimal, it expects the json pretty-print format from HF models.
2) The new model binary has the required data integrated, so no more additional files from there on.
Updating
By using the updated falcon quantizer you will convert the old model (best to start from 32 bit, 16 bit should be fine too) into a GGCC model (file version 10). You will need the falcon 40b tokenizer.json in the model directory.
https://huggingface.co/tiiuae/falcon-7b/blob/main/tokenizer.json
The new solution will need more tests, I am merging it anyway so this moves ahead.
I was not able to find any differences between this one and the transformers tokenizer, though given the scale of this addition there can be issues remaining, in worst case crashes. I ran the tokenizer on a couple random files and that worked fine.
This push addresses a serious difference in the tokenization, the one we used was the sentencepiece llama tokenizer. It is not able to tokenizer for Falcon, almost any long word was broken up (including special words) and international content was almost always complete garbage afterward. This is a serious issue to model quality, so it had to go.
This is a large change, I've spent 20+ hours first reversing what BPE in GPT-2 regex mode really does, then reimplementing it in C++ without the use of regular expressions and unicode libraries. Could have used regexp but it felt wrong to add such a chunky slowness into it. The whole task felt like reinventing the wheel while getting punched constantly. So I made a slim unicode library to work with and identify unicode data, bpe-decoders and -encoders similar to the special GPT2-BPE ones, a lookahead parser that should perform identical to the GPT/Falcon regular expression:
's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+.In addition I created a new magic and file version for Falcon ggllm.cpp called "GGCC" V0, there is no way around that now as the falcon tokenizer requires 65,000 merging sub-token pairs to operate. The model loader supports two modes: 1) old compatibility mode (also for quantizing/converting to GGCC) - you need to have the original tokenizer.json file in the same directory as the model binary. The original falcon-7b tokenizer.json should work for all finetunes because they do not change bpe merging. I did not integrate it into the python code, it's still creating llama ggml V1 to be converted. The json parser I wrote is absolute minimal, it expects the json pretty-print format from HF models. 2) The new model binary has the required data integrated, so no more additional files from there on.
Updating By using the updated falcon quantizer you will convert the old model (best to start from 32 bit, 16 bit should be fine too) into a GGCC model (file version 10). You will need the falcon 40b tokenizer.json in the model directory. https://huggingface.co/tiiuae/falcon-7b/blob/main/tokenizer.json
The new solution will need more tests, I am merging it anyway so this moves ahead. I was not able to find any differences between this one and the transformers tokenizer, though given the scale of this addition there can be issues remaining, in worst case crashes. I ran the tokenizer on a couple random files and that worked fine.
Test input 1:
Result now (identical to transformers tokenization):
Handling of fine tuned custom special tokens: Before:
New tokenizer: