search

dottxt-ai / outlines

Structured Text Generation
https://dottxt-ai.github.io/outlines/
Apache License 2.0
9.57k stars 490 forks source link

fix: Make `convert_token_to_string` pickleable #1236

Open saattrupdan opened 3 weeks ago

saattrupdan commented 3 weeks ago

This moves the convert_token_to_string helper function out as an independent function, to make it pickleable.

This is causing issues when storing logits processors in a class that needs to be pickleable.

saattrupdan commented 1 week ago

@rlouf, does this seem good to you?