[BUG]: connection aborted when running the example experiment

[zhenwu0831]

I was running the example experiment, but my docker log gives me two errors: 1) failed loading redisAI API; 2) possible security attack detected such that my connection is aborted. The steps that I followed are as follows.

1. setup I'm running sotopia locally on Mac and using docker. I set my redis om url as conda env config vars set REDIS_OM_URL="redis://:QzmCUD3C3RdsR@localhost:6379"

2. data I followed the instructions in Q&A using these commands: mkdir ~/redis-data curl -L https://cmu.box.com/shared/static/xiivc5z8rnmi1zr6vmk1ohxslylvynur --output ~/redis-data/dump.rdb sudo docker run -d -e REDIS_ARGS="--requirepass QzmCUD3C3RdsR" --name redis-stack-server -p 6379:6379 -p 8001:8001 -v ~/redis-data:/data redis/redis-stack-server:latest

3. run the example experiment python examples/experiment_eval.py --gin_file sotopia_conf/generation_utils_conf/generate.gin --gin_file sotopia_conf/server_conf/server.gin --gin_file sotopia_conf/run_async_server_in_batch.gin '--gin.ENV_IDS=["01H7VFHPDZVVCDZR3AARA547CY"]' '--gin.AGENT1_MODEL="gpt-4"' '--gin.BATCH_SIZE=20' '--gin.PUSH_TO_DB=False' '--gin.TAG="test"'

Additionally, it seems like the easy sample server running command raises not implemented error b/c of the missing implementation in samplers/base_sampler.py?

Thank you so much!

[ProKil]

Hi @zhenwu0831 thanks for your issue.

Additionally, it seems like the easy sample server running command raises not implemented error b/c of the missing implementation in samplers/base_sampler.py?

I can confirm that the readme command is out of date. For now, you can set the default sampler to UniformSampler. I will think of a way to fix this.

1) failed loading redisAI API; 2) possible security attack detected such that my connection is aborted. The steps that I followed are as follows.

I have never seen this error before. Could you try to set it up on a linux server? I think it should work for mac since in Issue #7 there were other users confirming that they succeeded in setting up the Redis server on mac OS machines.

[zhenwu0831]

Thank you so much @ProKil , I gave mac another try and I think I probably figured out the previous issue. When I go to localhost:8001, I can see the database so good. But still when I run the experiment command, the environment is not running. Below is the terminal output:

command:

python examples/experiment_eval.py --gin_file sotopia_conf/generation_utils_conf/generate.gin --gin_file sotopia_conf/server_conf/server.gin --gin_file sotopia_conf/run_async_server_in_batch.gin '--gin.ENV_IDS=["01H7VFHPDZVVCDZR3AARA547CY"]' '--gin.AGENT1_MODEL="gpt-4"' '--gin.BATCH_SIZE=20' '--gin.PUSH_TO_DB=False' '--gin.TAG="test"'

output:

Rewritten gin arg: --gin_bindings=ENV_IDS = ["01H7VFHPDZVVCDZR3AARA547CY"] Rewritten gin arg: --gin_bindings=AGENT1_MODEL = "gpt-4" Rewritten gin arg: --gin_bindings=BATCH_SIZE = 20 Rewritten gin arg: --gin_bindings=PUSH_TO_DB = False Rewritten gin arg: --gin_bindings=TAG = "test" [19:10:41] INFO [19:10:41] - INFO - root - Gin Configuration: gin_utils.py:55 I0204 19:10:41.844543 7950868160 gin_utils.py:55] Gin Configuration: INFO [19:10:41] - INFO - root - from gin import dynamic_registration gin_utils.py:57 I0204 19:10:41.846666 7950868160 gin_utils.py:57] from gin import dynamic_registration INFO [19:10:41] - INFO - root - import main as train_script gin_utils.py:57 I0204 19:10:41.847162 7950868160 gin_utils.py:57] import main as train_script INFO [19:10:41] - INFO - root - import sotopia.envs.evaluators as evaluators gin_utils.py:57 I0204 19:10:41.847525 7950868160 gin_utils.py:57] import sotopia.envs.evaluators as evaluators INFO [19:10:41] - INFO - root - import sotopia.envs.parallel as parallel gin_utils.py:57 I0204 19:10:41.847877 7950868160 gin_utils.py:57] import sotopia.envs.parallel as parallel INFO [19:10:41] - INFO - root - import sotopia.generation_utils.generate as generate gin_utils.py:57 I0204 19:10:41.848448 7950868160 gin_utils.py:57] import sotopia.generation_utils.generate as generate INFO [19:10:41] - INFO - root - import sotopia.server as server gin_utils.py:57 I0204 19:10:41.848789 7950868160 gin_utils.py:57] import sotopia.server as server INFO [19:10:41] - INFO - root - gin_utils.py:57 I0204 19:10:41.849128 7950868160 gin_utils.py:57] INFO [19:10:41] - INFO - root - # Macros: gin_utils.py:57 I0204 19:10:41.849452 7950868160 gin_utils.py:57] # Macros: INFO [19:10:41] - INFO - root - # gin_utils.py:57 ==============================================================================
I0204 19:10:41.849760 7950868160 gin_utils.py:57] # ============================================================================== INFO [19:10:41] - INFO - root - ACTION_ORDER = 'round-robin' gin_utils.py:57 I0204 19:10:41.850128 7950868160 gin_utils.py:57] ACTION_ORDER = 'round-robin' INFO [19:10:41] - INFO - root - AGENT1_MODEL = 'gpt-4' gin_utils.py:57 I0204 19:10:41.850466 7950868160 gin_utils.py:57] AGENT1_MODEL = 'gpt-4' INFO [19:10:41] - INFO - root - AGENT2_MODEL = 'gpt-3.5-turbo' gin_utils.py:57 I0204 19:10:41.850780 7950868160 gin_utils.py:57] AGENT2_MODEL = 'gpt-3.5-turbo' INFO [19:10:41] - INFO - root - BATCH_SIZE = 20 gin_utils.py:57 I0204 19:10:41.851115 7950868160 gin_utils.py:57] BATCH_SIZE = 20 INFO [19:10:41] - INFO - root - ENV_IDS = ['01H7VFHPDZVVCDZR3AARA547CY'] gin_utils.py:57 I0204 19:10:41.851455 7950868160 gin_utils.py:57] ENV_IDS = ['01H7VFHPDZVVCDZR3AARA547CY'] INFO [19:10:41] - INFO - root - LITE = False gin_utils.py:57 I0204 19:10:41.851809 7950868160 gin_utils.py:57] LITE = False INFO [19:10:41] - INFO - root - MAX_RETRIES = 6 gin_utils.py:57 I0204 19:10:41.852143 7950868160 gin_utils.py:57] MAX_RETRIES = 6 INFO [19:10:41] - INFO - root - MODEL_NAMES = {'agent1': %AGENT1_MODEL, 'agent2': gin_utils.py:57 %AGENT2_MODEL, 'env': 'gpt-4'}
I0204 19:10:41.852447 7950868160 gin_utils.py:57] MODEL_NAMES = {'agent1': %AGENT1_MODEL, 'agent2': %AGENT2_MODEL, 'env': 'gpt-4'} INFO [19:10:41] - INFO - root - OMNISCIENT = False gin_utils.py:57 I0204 19:10:41.852837 7950868160 gin_utils.py:57] OMNISCIENT = False INFO [19:10:41] - INFO - root - PUSH_TO_DB = False gin_utils.py:57 I0204 19:10:41.853148 7950868160 gin_utils.py:57] PUSH_TO_DB = False INFO [19:10:41] - INFO - root - TAG = 'test' gin_utils.py:57 I0204 19:10:41.853451 7950868160 gin_utils.py:57] TAG = 'test' INFO [19:10:41] - INFO - root - TAG_TO_CHECK_EXISTING_EPISODES = None gin_utils.py:57 I0204 19:10:41.853744 7950868160 gin_utils.py:57] TAG_TO_CHECK_EXISTING_EPISODES = None INFO [19:10:41] - INFO - root - VERBOSE = False gin_utils.py:57 I0204 19:10:41.854062 7950868160 gin_utils.py:57] VERBOSE = False INFO [19:10:41] - INFO - root - gin_utils.py:57 I0204 19:10:41.854354 7950868160 gin_utils.py:57] INFO [19:10:41] - INFO - root - # Parameters for evaluators.ReachGoalLLMEvaluator.acall: gin_utils.py:57 I0204 19:10:41.854628 7950868160 gin_utils.py:57] # Parameters for evaluators.ReachGoalLLMEvaluator.acall: INFO [19:10:41] - INFO - root - # gin_utils.py:57 ==============================================================================
I0204 19:10:41.854969 7950868160 gin_utils.py:57] # ============================================================================== INFO [19:10:41] - INFO - root - evaluators.ReachGoalLLMEvaluator.acall.temperature = 0.0 gin_utils.py:57 I0204 19:10:41.855298 7950868160 gin_utils.py:57] evaluators.ReachGoalLLMEvaluator.acall.temperature = 0.0 INFO [19:10:41] - INFO - root - gin_utils.py:57 I0204 19:10:41.855632 7950868160 gin_utils.py:57] INFO [19:10:41] - INFO - root - # Parameters for gin_utils.py:57 train_script._iterate_env_agent_combo_not_in_db:
I0204 19:10:41.855939 7950868160 gin_utils.py:57] # Parameters for train_script._iterate_env_agent_combo_not_in_db: INFO [19:10:41] - INFO - root - # gin_utils.py:57 ==============================================================================
I0204 19:10:41.856321 7950868160 gin_utils.py:57] # ============================================================================== INFO [19:10:41] - INFO - root - train_script._iterate_env_agent_combo_not_in_db.env_ids = gin_utils.py:57 %ENV_IDS
I0204 19:10:41.856680 7950868160 gin_utils.py:57] train_script._iterate_env_agent_combo_not_in_db.env_ids = %ENV_IDS INFO [19:10:41] - INFO - root - train_script._iterate_env_agent_combo_not_in_db.tag = \ gin_utils.py:57 I0204 19:10:41.858269 7950868160 gin_utils.py:57] train_script._iterate_env_agent_combo_not_in_db.tag = \ INFO [19:10:41] - INFO - root - %TAG_TO_CHECK_EXISTING_EPISODES gin_utils.py:57 I0204 19:10:41.859491 7950868160 gin_utils.py:57] %TAG_TO_CHECK_EXISTING_EPISODES INFO [19:10:41] - INFO - root - gin_utils.py:57 I0204 19:10:41.860324 7950868160 gin_utils.py:57] INFO [19:10:41] - INFO - root - # Parameters for generate.agenerate_action: gin_utils.py:57 I0204 19:10:41.860702 7950868160 gin_utils.py:57] # Parameters for generate.agenerate_action: INFO [19:10:41] - INFO - root - # gin_utils.py:57 ==============================================================================
I0204 19:10:41.861063 7950868160 gin_utils.py:57] # ============================================================================== INFO [19:10:41] - INFO - root - generate.agenerate_action.temperature = 1.0 gin_utils.py:57 I0204 19:10:41.861435 7950868160 gin_utils.py:57] generate.agenerate_action.temperature = 1.0 INFO [19:10:41] - INFO - root - gin_utils.py:57 I0204 19:10:41.862349 7950868160 gin_utils.py:57] INFO [19:10:41] - INFO - root - # Parameters for generate.agenerate_env_profile: gin_utils.py:57 I0204 19:10:41.862787 7950868160 gin_utils.py:57] # Parameters for generate.agenerate_env_profile: INFO [19:10:41] - INFO - root - # gin_utils.py:57 ==============================================================================
I0204 19:10:41.863184 7950868160 gin_utils.py:57] # ============================================================================== INFO [19:10:41] - INFO - root - generate.agenerate_env_profile.model_name = 'gpt-3.5-turbo' gin_utils.py:57 I0204 19:10:41.864560 7950868160 gin_utils.py:57] generate.agenerate_env_profile.model_name = 'gpt-3.5-turbo' INFO [19:10:41] - INFO - root - gin_utils.py:57 I0204 19:10:41.865566 7950868160 gin_utils.py:57] INFO [19:10:41] - INFO - root - # Parameters for generate.obtain_chain: gin_utils.py:57 I0204 19:10:41.866327 7950868160 gin_utils.py:57] # Parameters for generate.obtain_chain: INFO [19:10:41] - INFO - root - # gin_utils.py:57 ==============================================================================
I0204 19:10:41.866681 7950868160 gin_utils.py:57] # ============================================================================== INFO [19:10:41] - INFO - root - generate.obtain_chain.max_retries = %MAX_RETRIES gin_utils.py:57 I0204 19:10:41.867256 7950868160 gin_utils.py:57] generate.obtain_chain.max_retries = %MAX_RETRIES INFO [19:10:41] - INFO - root - gin_utils.py:57 I0204 19:10:41.867653 7950868160 gin_utils.py:57] INFO [19:10:41] - INFO - root - # Parameters for parallel.ParallelSotopiaEnv.reset: gin_utils.py:57 I0204 19:10:41.867959 7950868160 gin_utils.py:57] # Parameters for parallel.ParallelSotopiaEnv.reset: INFO [19:10:41] - INFO - root - # gin_utils.py:57 ==============================================================================
I0204 19:10:41.868309 7950868160 gin_utils.py:57] # ============================================================================== INFO [19:10:41] - INFO - root - parallel.ParallelSotopiaEnv.reset.lite = %LITE gin_utils.py:57 I0204 19:10:41.868670 7950868160 gin_utils.py:57] parallel.ParallelSotopiaEnv.reset.lite = %LITE INFO [19:10:41] - INFO - root - gin_utils.py:57 I0204 19:10:41.869040 7950868160 gin_utils.py:57] INFO [19:10:41] - INFO - root - # Parameters for server.run_async_server: gin_utils.py:57 I0204 19:10:41.869364 7950868160 gin_utils.py:57] # Parameters for server.run_async_server: INFO [19:10:41] - INFO - root - # gin_utils.py:57 ==============================================================================
I0204 19:10:41.869714 7950868160 gin_utils.py:57] # ============================================================================== INFO [19:10:41] - INFO - root - server.run_async_server.action_order = %ACTION_ORDER gin_utils.py:57 I0204 19:10:41.870073 7950868160 gin_utils.py:57] server.run_async_server.action_order = %ACTION_ORDER INFO [19:10:41] - INFO - root - server.run_async_server.model_dict = \ gin_utils.py:57 I0204 19:10:41.870429 7950868160 gin_utils.py:57] server.run_async_server.model_dict = \ INFO [19:10:41] - INFO - root - {'agent1': 'gpt-3.5-turbo', 'agent2': 'gpt-3.5-turbo', gin_utils.py:57 'env': 'gpt-4'}
I0204 19:10:41.870879 7950868160 gin_utils.py:57] {'agent1': 'gpt-3.5-turbo', 'agent2': 'gpt-3.5-turbo', 'env': 'gpt-4'} INFO [19:10:41] - INFO - root - server.run_async_server.omniscient = %OMNISCIENT gin_utils.py:57 I0204 19:10:41.871398 7950868160 gin_utils.py:57] server.run_async_server.omniscient = %OMNISCIENT INFO [19:10:41] - INFO - root - server.run_async_server.push_to_db = %PUSH_TO_DB gin_utils.py:57 I0204 19:10:41.871802 7950868160 gin_utils.py:57] server.run_async_server.push_to_db = %PUSH_TO_DB INFO [19:10:41] - INFO - root - server.run_async_server.tag = %TAG gin_utils.py:57 I0204 19:10:41.872172 7950868160 gin_utils.py:57] server.run_async_server.tag = %TAG INFO [19:10:41] - INFO - root - server.run_async_server.using_async = True gin_utils.py:57 I0204 19:10:41.872519 7950868160 gin_utils.py:57] server.run_async_server.using_async = True INFO [19:10:41] - INFO - root - gin_utils.py:57 I0204 19:10:41.872891 7950868160 gin_utils.py:57] INFO [19:10:41] - INFO - root - # Parameters for train_script.run_async_server_in_batch: gin_utils.py:57 I0204 19:10:41.873193 7950868160 gin_utils.py:57] # Parameters for train_script.run_async_server_in_batch: INFO [19:10:41] - INFO - root - # gin_utils.py:57 ==============================================================================
I0204 19:10:41.873564 7950868160 gin_utils.py:57] # ============================================================================== INFO [19:10:41] - INFO - root - train_script.run_async_server_in_batch.batch_size = gin_utils.py:57 %BATCH_SIZE
I0204 19:10:41.873920 7950868160 gin_utils.py:57] train_script.run_async_server_in_batch.batch_size = %BATCH_SIZE INFO [19:10:41] - INFO - root - train_script.run_async_server_in_batch.model_names = gin_utils.py:57 %MODEL_NAMES
I0204 19:10:41.874326 7950868160 gin_utils.py:57] train_script.run_async_server_in_batch.model_names = %MODEL_NAMES INFO [19:10:41] - INFO - root - train_script.run_async_server_in_batch.tag = %TAG gin_utils.py:57 I0204 19:10:41.874729 7950868160 gin_utils.py:57] train_script.run_async_server_in_batch.tag = %TAG INFO [19:10:41] - INFO - root - train_script.run_async_server_in_batch.verbose = %VERBOSE gin_utils.py:57 I0204 19:10:41.875083 7950868160 gin_utils.py:57] train_script.run_async_server_in_batch.verbose = %VERBOSE Running all envs in batch: 0it [00:00, ?it/s]

[ProKil]

What do you by

the environment is not running

?

If the problem is no environments are found, then it probably means that you have not loaded the available tasks into the database. As a quick check, can you tell me the output of the following lines:

from sotopia.database import EnvironmentProfile
len(list(EnvironmentProfile.all_pks()))

[zhenwu0831]

The output is 90.

It looks to me that the environment is set up but hasn't completed any iterations yet... Would really appreciate it if I could get some help on this, thanks!

[ProKil]

OK I know what's happening. The database you downloaded already has some episodes in it for the env id you used, so it automatically prevents new episodes from generating. What about the following command?

python examples/experiment_eval.py --gin_file sotopia_conf/generation_utils_conf/generate.gin --gin_file sotopia_conf/server_conf/server.gin --gin_file sotopia_conf/run_async_server_in_batch.gin '--gin.ENV_IDS=["01H7VFHPDZVVCDZR3AARA547CY"]' '--gin.AGENT1_MODEL="gpt-4"' '--gin.BATCH_SIZE=20' '--gin.PUSH_TO_DB=False' '--gin.TAG="test"' '--gin.TAG_TO_CHECK_EXISTING_EPISODES="test"'

[zhenwu0831]

Thank you so much! Now I can see the episodes and the evaluation in my terminal output :) I also want to confirm if the output is only console-based or if graphical user interface is available?

[ProKil]

No the GUI is only for running interactive experiments with humans.

[ProKil]

@zhenwu0831 would you mind sharing how you solved the Redis DB bug?

[zhenwu0831]

Hi @ProKil Thank you! I'm wondering if I could run the interactive experiments with humans on my end? In /sotopia_conf, I could only find the asynchronous server gin files...

For the redis issue, it turns out to be a localhost:6379 open in my background and I didn't notice it lol