search

stanford-crfm / helm

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in Holistic Evaluation of Text-to-Image Models (HEIM) (https://arxiv.org/abs/2311.04287).
https://crfm.stanford.edu/helm
Apache License 2.0
1.87k stars 243 forks source link

o1 series models cannot take system prompt #3019

Open bryanzhou008 opened 2 hours ago

bryanzhou008 commented 2 hours ago

It seems that the current o1 series models o1-mini-2024-09-12 and o1-preview-2024-09-12 are still not able to take system prompts. Would it be a good idea to set them to concatenate system prompts with user prompts? Thanks!

yifanmai commented 2 hours ago

I don't think so, because the model API should reflect the actual OpenAI API as closely as possible. Rewriting the request internally can lead to unexpected results for other researchers.