[Question] Empty string returned when using google/gemma-2-9b-it model of Together AI API

[yhg8423]

Hello,

I recently went through the agent_development.ipynb tutorial. I used API_TYPE='together_ai', MODEL_NAME='google/gemma-2-9b-it' as recommended in the tutorial. I noticed that during the runnable_simulation() process, LLM generates an empty string (''). In the api response of together_ai, I found that result = response.choices[0].message.content returns an empty string (''). Is there any way to fix this?

Thank you!

[vezhnick]

Hello!

In which context is the call to llm made? It could be that the call had terminators = ['\n'] and llm responded with a newline and was cut off?

Could you share more of the code you are running?

[yhg8423]

Thanks for your quick response!

I am currently running my_agent.py from agent_development.ipynb with colab. First, I got warning message WARNING:concordia.associative_memory.formative_memories:Warning: Number of generated formative episodes (8) does not match number of formative ages (7). This is just a warning and probably not problematic. when execute "Initialize the simulation" section as below.

runnable_simulation = scenarios.build_simulation(
    scenarios.SCENARIO_CONFIGS[SCEANRIO_NAME],
    model=model,
    embedder=embedder,
    measurements=measurements,
    focal_agent_module=agent_module,
    override_agent_model=call_limit_wrapper.CallLimitLanguageModel(model),
) 

After that, during the running simulation with simulation_outcome, results_log = runnable_simulation(), the log shows only prefix such as agent's name although I use a language model. Below is a portion of the output log.

Rowan Cranberry's attempted action: Rowan Cranberry 

Rowan Cranberry 
Current state of conversation: Here is the conversation from the beginning:
Rowan Cranberry 
Hunter Nutwood 
Rowan Cranberry 
GM context of action and chain of thought:
Instructions: This is a social science experiment. It is structured as a tabletop roleplaying game (like dungeons and dragons). You are the game master. You will describe the current situation to the participants in the experiment and then on the basis of what you tell them they will suggest actions for the character they control. Aside from you, each other participant controls just one character. You are the game master so you may control any non-player character. You will track the state of the world and keep it consistent as time passes in the simulation and the participants take actions and change things in their world. Remember that this is a serious social science experiment. It is not just a game. It need not be fun for the participants. Always use third-person limited perspective, even when speaking directly to the participants.

Conversation history: Here is the conversation from the beginning:
Rowan Cranberry 
Hunter Nutwood 
Rowan Cranberry 

I think there should be an output generated based on the language model next to the agent name, but it doesn't generate any output as shown above!

[vezhnick]

Hi! What is most likely is that call_limit_wrapper.CallLimitLanguageModel has reached the limit of calls. You can simply disable it by doing removing this line: override_agent_model=call_limit_wrapper.CallLimitLanguageModel(model)

[yhg8423]

Thank you for response!

Unfortunately, I tried removing the line override_agent_model=call_limit_wrapper.CallLimitLanguageModel(model), but I still have the same problem. Therefore, I printed response.choices[0].message in language/together_ai.py, I founded that the response of llm sometimes generate only empty string such as below two lines.

role=<MessageRole.ASSISTANT: 'assistant'> content='\n\n' tool_calls=[]
role=<MessageRole.ASSISTANT: 'assistant'> content='' tool_calls=[]
role=<MessageRole.ASSISTANT: 'assistant'> content='' tool_calls=[]

[jzleibo]

Is the issue that it always generates empty strings or that it sometimes generates empty strings? Do you ever see normal responses from this LLM?

[yhg8423]

@jzleibo Thank you for the response!

I am not sure, but I think It always generates empty strings, because I do not see normal responses from the LLM in output logs and HTML logs (below image is an example image of HTML log's conversation scene).

I also print the response.choices[0].message in together_ai.py as above. I can see below printed results.

role=<MessageRole.ASSISTANT: 'assistant'> content='' tool_calls=[] role=<MessageRole.ASSISTANT: 'assistant'> content=' ' tool_calls=[] role=<MessageRole.ASSISTANT: 'assistant'> content='[16 Sep 1895 00:00:00] [observation] The cobblestone streets of Fruitville echoed with the clatter of hooves and the rumble of carts as the market awoke. Underneath colorful awnings, merchants proudly presented their bountiful harvests, their voices a chorus of greetings and bartering. Hunter Nutwood is trying to buy some fruit from Blaise Windwillow. They are negotiating a price. Hunter Nutwood can sell the fruit for 5 coins back in her home town.\n[16 Sep 1895 00:10:00] [observation] Hunter Nutwood is ready to make an offer.\n\n\n' tool_calls=[] role=<MessageRole.ASSISTANT: 'assistant'> content='' tool_calls=[] role=<MessageRole.ASSISTANT: 'assistant'> content='' tool_calls=[] role=<MessageRole.ASSISTANT: 'assistant'> content='' tool_calls=[] role=<MessageRole.ASSISTANT: 'assistant'> content='' tool_calls=[] role=<MessageRole.ASSISTANT: 'assistant'> content='[16 Sep 1895 00:00:00] The cobblestone streets of Fruitville echoed with the clatter of hooves and the rumble of carts as the market awoke. Underneath colorful awnings, merchants proudly presented their bountiful harvests, their voices a chorus of greetings and bartering. Blaise Windwillow is trying to sell some fruit. He is negotiating a price with Hunter Nutwood. It costs Blaise Windwillow 2 coin to buy the fruit from the farm.\n[16 Sep 1895 00:10:00] Blaise Windwillow has to accept or reject the offer.\n[16 Sep 1895 00:10:00] Hunter Nutwood proposed 2 coins. \n\n' tool_calls=[] role=<MessageRole.ASSISTANT: 'assistant'> content='' tool_calls=[] role=<MessageRole.ASSISTANT: 'assistant'> content='' tool_calls=[]

[vezhnick]

Hey! We reproduced the issue and looking into it

[yhg8423]

Oh, I see! Thanks!

[kevin-v96]

Any updates on this? @vezhnick

[jzleibo]

The gemma-2-9b-it model from together.ai is still broken, that's why it is returning empty strings. We are in contact with someone from together.ai now, and trying to get them to fix it as quickly as possible. In the meantime, this should not be blocking for work on the Concordia contest since the issue is specific to the 9b model. We recommend using gemma-2-27b-it for now, until together.ai fixes the 9b model.

For now, the plan is still to use gemma-2-9b-it for the final evaluation. But if the current issue drags on much longer then we will have to re-evaluate that. If we do end up switching away from that model then the most likely alternative will be gemma-2-27b-it.

[jzleibo]

We are updating the contest Slack with the latest as this goes on: https://join.slack.com/t/neuripsconcor-zjd8720/shared_invite/zt-2pvn5a6k3-rL6QUqxD1uuYy6gIGNbByw