joonspk-research / generative_agents

Generative Agents: Interactive Simulacra of Human Behavior
Apache License 2.0
17.46k stars 2.25k forks source link

Prompt is unclear what should be picked from a list of choices #85

Open edwios opened 1 year ago

edwios commented 1 year ago

First of all, this is a really fun and inspiring simulation, thank you for all the efforts making this great start.

After adapting the simulation to use a local LLM (llama.cpp), I have identified a potential issue about picking items from a list of objects or destinations.

For example, in the file action_location_object_vMar11.txt, the part involving picking one of the locations from the list of areas was written like: !<INPUT 3>! is !<INPUT 4>!. For !<INPUT 5>!, !<INPUT 6>! should go to the following area in !<INPUT 7>! (MUST pick one of {!<INPUT 8>!}):

With this, there is a chance the LLM would pick a more generic term out of these selections instead of the full name of the item. For example, with areas of ["man's bathroom", "woman's bathroom", "common room", "kitchen"], the LLM might just return "bathroom" instead of "man's bathroom" or "woman's bathroom".

To minimise the chance of this from happening, adding further clarifications like "a comma separated list" would encourage the LLM to return the full name of the item. For example: !<INPUT 3>! is !<INPUT 4>!. For !<INPUT 5>!, !<INPUT 6>! should go to the following area in !<INPUT 7>! (MUST pick one item from this comma separated list {!<INPUT 8>!}. You must include all the words of that item.):

bellowswang commented 1 year ago

I have a similar error. I think in my case main room is a too generic term probably, so when the LLM heard that it should pick from {main room}, it just answers kitchen...

Isabella Rodriguez is going to Isabella Rodriguez's apartment that has the following areas: {main room}
* Stay in the current area if the activity can be done there.
* NEVER go into other people's rooms unless necessary.
Isabella Rodriguez is sleeping. For sleeping, Isabella Rodriguez should go to the following area in Isabella Rodriguez's apartment (MUST pick one of {main room}):
Answer: {

~~~ output    ----------------------------------------------------
kitchen

=== END ==========================================================

GNS FUNCTION: <generate_action_game_object>
Traceback (most recent call last):
  File "/Users/yihongwang/Projects/mcg/generative_agents/reverie/backend_server/persona/memory_structures/spatial_memory.py", line 105, in get_str_accessible_arena_game_objects
    x = ", ".join(list(self.tree[curr_world][curr_sector][curr_arena]))
KeyError: 'kitchen'
edwios commented 1 year ago

Fortunately, a better tuned prompt would solve this.