Using TogetherChatClient instead of TogetherClient for the Llama-3 Chat models

stanford-crfm / helm

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in Holistic Evaluation of Text-to-Image Models (HEIM) (https://arxiv.org/abs/2311.04287).

Apache License 2.0

1.89k stars 244 forks source link

When I tried to run the together/llama-3-8b-chat and together/llama-3-70b-chat models with the default setup in src/helm/config/model_deployments.yaml, both models have client_spec set to "helm.clients.together_client.TogetherClient". However, this leads to many strange behaviors at generation time, such as repetition of instructions or outputting "..." as the answer.

This behavior stopped after I updated src/helm/config/model_deployments.yaml to reset the client_spec to "helm.clients.together_client.TogetherChatClient". I wonder whether this is a better client to handle these chat models and perhaps should be the default?

Thank!

stanford-crfm / helm

Using TogetherChatClient instead of TogetherClient for the Llama-3 Chat models #2693