search

robertknight / rten

ONNX neural network inference engine
124 stars 9 forks source link

Add `token_id_filter` function to create logits filter from predicate #394

Closed robertknight closed 3 weeks ago

robertknight commented 3 weeks ago

A very common use case for logit filters is to prevent certain token IDs from being sampled. Add a token_id_filter helper for this.

In the process Generator was modified to support samplers and logit filters with non-static lifetimes.