sotopia-lab / sotopia

Sotopia: an Open-ended Social Learning Environment (ICLR 2024 spotlight)
https://docs.sotopia.world
MIT License
165 stars 20 forks source link

[BUG]: experimental_eval.py cannot be successfully run with 0.0.11 or 0.1.0-rcx #200

Open lwaekfjlk opened 1 month ago

lwaekfjlk commented 1 month ago

Description of the bug

class name including EvaluationForTwoAgents is not supported

Steps To Reproduce

run experimental_eval.py with sotopia==0.0.11

independently install sotopia

python experimental_eval.py \ --gin_file sotopia_conf/generation_utils_conf/generate.gin \ --gin_file sotopia_conf/server_conf/server.gin \ --gin_file sotopia_conf/run_async_server_in_batch.gin \ '--gin.ENV_IDS=["01H7VFHNV13MHN97GAH73E3KM8", "01H7VFHN5WVC5HKKVBHZBA553R", "01H7VFHNN7XTR99319DS8KZCQM", "01H7VFHN9W0WAFZCBT09PKJJNK", "01H7VFHPDZVVCDZR3AARA547CY", "01H7VFHPQQQY6H4DNC6NBQ8XTG", "01H7VFHPQQQY6H4DNC6NBQ8XTG", "01H7VFHN7WJK7VWVRZZTQ6DX9T", "01H7VFHN7A1ZX5KSMT2YN9RXC4", "01H7VFHPS5WJW2694R1MNC8JFY", "01H7VFHPS5WJW2694R1MNC8JFY", "01H7VFHNN7XTR99319DS8KZCQM", "01H7VFHQ11NAMZS4A2RDGDB01V", "01H7VFHQ11NAMZS4A2RDGDB01V", "01H7VFHPSWGDGEYRP63H2DJKV0", "01H7VFHPSWGDGEYRP63H2DJKV0", "01H7VFHNF4G18PC9JHGRC8A1R6", "01H7VFHNNYH3W0VRWVY178K2TK", "01H7VFHP8AN5643B0NR0NP00VE", "01H7VFHN7A1ZX5KSMT2YN9RXC4"]' \ '--gin.AGENT1_MODEL="gpt-3.5-turbo"' \ '--gin.AGENT2_MODEL="gpt-3.5-turbo"' \ '--gin.BATCH_SIZE=1' \ '--gin.PUSH_TO_DB=True' \ '--gin.TAG="gpt-3.5-turbo_vs_gpt-3.5-turbo-0927"'

Additional Information

No response

XuhuiZhou commented 1 month ago

@lwaekfjlk is this fixed or still there? imo it's bc we removed the retrieved prompt there and we are still figuring out a way to fix this.

lwaekfjlk commented 1 month ago

still there I believe