search

stanford-crfm / helm

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in Holistic Evaluation of Text-to-Image Models (HEIM) (https://arxiv.org/abs/2311.04287).
https://crfm.stanford.edu/helm
Apache License 2.0
1.77k stars 235 forks source link

Set stop sequences to empty for MedLM-large #2709

Closed yifanmai closed 1 month ago

yifanmai commented 1 month ago

MedLM-large does not support stop sequences, even though MedLM-medium supports stop sequences. We need to remove all stop sequences for this model.

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "Unable to submit request because the model does not support the stopSequences parameter. Remove it and try again. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/overview"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:xxx.xxx.xxx.xxx:443 {grpc_message:"Unable to submit request because the model does not support the stopSequences parameter. Remove it and try again. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/overview", grpc_status:3, created_time:"2024-06-03T15:52:44.236371716-07:00"}"
>