duplicate tokens in tokenizers

microsoft / aici

AICI: Prompts as (Wasm) Programs

MIT License

1.87k stars 76 forks source link

duplicate tokens in tokenizers #78

Open mmoskal opened 4 months ago

mmoskal commented 4 months ago

For example, the llama tokenizer has "<0x20>" as 35 and "▁" (space) as 29871, as well as "<0x21>" as 36 and "!" as 29991, etc.

We need to:

pick the canonical form (29871 probably)
have a mapping on the side that if 29871 is allowed also allows 35 in TokenSet (apply it after "compute_bias()" etc).

mmoskal commented 4 months ago

mostly done, need to call apply_duplicates() in more places in particular somewhere around return_logit_bias() and possibly after any user-level update to token set