Guided choice not respected

[Andrea-de-Varda]

Describe the bug The guided_choice option in llmengine.Completion.create() is not respected in the API call. The response ignores the guided choices I specify, regardless of the model I'm considering, and produces its usual completion output. I installed in a clean environment.

LLM Engine Version

• LLM Engine Version: 0.0.0beta39

System Version

• Python Version: 3.10.12
• Operating System: Ubuntu 22.04.4 LTS

Timestamp and Request ID

• timestamp:
• request_id: ade60281-7feb-48ce-921b-d7d081e2a73d

Minimal Reproducible Example Steps to reproduce the behavior:

1. Install LLM Engine

   pip install scale-llm-engine

2. Make API call

   
   from llmengine import api_engine, Completion

api_engine.set_api_key("MASKED")

response = Completion.create( model="llama-2-7b", # The issues persists with all the models I tried prompt="Hello, my favourite", max_new_tokens=10, temperature=0, guided_choice=["food", "thing", "car"], )

print(response.json())

3. See error (the response text does not contain any of the  options in the `guided_choice ` list)

{"request_id": "ade60281-7feb-48ce-921b-d7d081e2a73d", "output": {"text": " time of the year is here again. I love", "num_prompt_tokens": null, "num_completion_tokens": 10, "tokens": null}}

**Expected behavior**
One of the tokens or string from the `guided_choice` list should have been returned

**Additional context**
Additionally, when I import, I get the message:

A newer version (0.0.0b39) of 'scale-llm-engine' is available. Please upgrade! To upgrade, run: pip install --upgrade scale-llm-engine Don't want to see this message? Set the environment variable 'LLM_ENGINE_DISABLE_VERSION_CHECK' to 'true'.

However, I am already using the latest version:

import llmengine print(llmengine.version) # returns 0.0.0beta39

[yixu34]

Hi @Andrea-de-Varda, thanks for filing this issue! We're triaging.

[yixu34]

Hi @Andrea-de-Varda , please see https://github.com/scaleapi/llm-engine/issues/619 - we do have this feature in the code, but we're sunsetting the free demo, which is out of date with what's in Github and what we run internally.