search

KerfuffleV2 / llm-samplers

A collection of LLM token samplers in Rust
MIT License
15 stars 4 forks source link

Possible contribution: Token Healing #6

Open JanIngoHaas opened 9 months ago

JanIngoHaas commented 9 months ago

Hi there! I could probably contribute a token healing sampler like it's done in guidance. Is this a feature that you would like to see?

KerfuffleV2 commented 9 months ago

Hi! I'd say for the most part any useful feature is very welcome.

I wasn't familiar with "token healing" but I read a little bit just now. I'm not sure if you would be able to add it here with the current interface. Currently samplers only have access to:

  1. The last generated tokens (but no way to tell which part is the initial prompt)
  2. The current logits.
  3. A random number generator.

When they run, they can only do two things: manipulate the logits and/or produce a token.

Do you think you can implement a token healing sampler within those limitations?

By the way, I'm actually really unhappy with the way resources (last tokens, logits, RNG) are implemented. I've been trying to figure out a better way to handle it without much success so far. I got a bit discouraged and haven't worked on this project for a while.