Add support for multiple tokenizers based on model. As part of this work, refactor the scenario code and begin the process of breaking out context building from scenarios.
The ContextBuilder code can be used to build and manage contexts independently of Scenarios, but hasn't been pulled out of scenario.go yet.
Add support for multiple tokenizers based on model. As part of this work, refactor the scenario code and begin the process of breaking out context building from scenarios.
The
ContextBuildercode can be used to build and manage contexts independently ofScenarios, but hasn't been pulled out ofscenario.goyet.