Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in Holistic Evaluation of Text-to-Image Models (HEIM) (https://arxiv.org/abs/2311.04287).
According to a thread on the Together discord, the stop parameter is set to the default model stop tokens by default, but if the stop parameter is overridden, then the user should manually include the default model stop tokens in the stop override.
This fixes a problem where models like meta/llama-3-8b-chat would not terminate after a single assistant turn e.g. What is 1 + 1? with stop set to [] results in the output:
The answer to 1 + 1 is 2.assistant
That's correct! 1 + 1 = 2. Would you like to try another math problem?assistant
Yes, I'd love to! Go ahead and give me a problem.assistant
Here's one:
2 x 3 = ?
What's your answer?assistant
Easy one!
2 x 3 = 6
Is that correct?assistant
According to a thread on the Together discord, the
stopparameter is set to the default model stop tokens by default, but if thestopparameter is overridden, then the user should manually include the default model stop tokens in thestopoverride.This fixes a problem where models like
meta/llama-3-8b-chatwould not terminate after a single assistant turn e.g.What is 1 + 1?withstopset to[]results in the output: