Compliance with standard gym API

[leobxpan]

Hi, it seems that the current repo does not fully comply with the standard gym API? E.g., if I create an env using gym.make('Overcooked-v0'), its action space will be None. Am I doing something wrong here?

[leobxpan]

Or alternatively, if I use the custom_init() function to create it, what should I pass in for featurize_fn?

[mesutyang97]

Could you post the backtrace?

Regarding your second question, you should pass in the featurize_fn of the embedded base_ae.env.featurize_state_mdp or base_ae.env.lossless_state_encoding_mdp

[leobxpan]

Thanks.

Creating Overcooked instance from gym.make():

>>> import gym
>>> env = gym.make('Overcooked-v0')
>>> print(env.action_space)
None

Basically, what is the expected way to create an Overcooked instance?

[mesutyang97]

I see where the confusion might come from.

We are actually using rllib to wrap our entire interface, and rllib takes care of initializing the Overcooked gym object, via the custom_init

@nathan-miller23 worked on this section of the codebase, and I believe he can answer these questions better than I do. But I have provided my best response.

[leobxpan]

Thanks @mesutyang97. However I didn't seem to find anywhere in the human_aware_rl repo that uses Overcooked class, rather it seems to directly use the lower level OvercookedEnv. I'm working with some interfaces that take a gym instance as input so I'd like to directly work with Overcooked. From my understanding, if I want to create one such instance directly using your API defined without using rllib, I should do the following:

mdp = OvercookedGridworld.from_layout_name("cramped_room_single")
base_env = OvercookedEnv.from_mdp(mdp)
env = gym.make('Overcooked-v0')
env = env.custom_init(base_env, base_env.featurize_state_mdp)

Please correct me if I'm wrong. Also, both featurize_state_mdp() and lossless_state_encoding_mdp() take only 2 arguments(including self) but on L567 it inputs 3 arguments (including self). More insights would be appreciated.

[mesutyang97]

I see. I tend to believe that is the right way to use the interface.

Hmm OK I think I see what you mean. It seems like we are indeed not using the gym interface in our current system, and this section of the code is outdated. Sorry we overlooked gym compatibility when we were upgrading the codebase over the summer.

I have a proposed fix (which you are welcome to try out): drop the mdp in all calls of featurize_fn. But I will work on this today and tomorrow to come up with a more comprehensive (and well-tested) fix for the gym compatibility.

Sorry about the hassle, and thanks for pointing out the issue.

[leobxpan]

No worries, and thanks for working on this.

I tried your method, but it gave out the following error:

>>> env = env.custom_init(base_env, base_env.featurize_state_mdp)
Computing MediumLevelActionManager
Recomputing planner due to: [Errno 2] No such file or directory: '/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/data/planners/cramped_room_single_am.pkl'
Computing MediumLevelActionManager to be saved in /data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/data/planners/cramped_room_single_am.pkl
Traceback (most recent call last):
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 801, in from_pickle_or_compute
    mlam = MediumLevelActionManager.from_file(filename)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 789, in from_file
    return load_saved_action_manager(filename)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/data/planners/__init__.py", line 6, in load_saved_action_manager
    with open(os.path.join(PLANNERS_DIR, filename), 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/data/planners/cramped_room_single_am.pkl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/mdp/overcooked_env.py", line 557, in custom_init
    self.observation_space = self._setup_observation_space()
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/mdp/overcooked_env.py", line 565, in _setup_observation_space
    obs_shape = self.featurize_fn(dummy_state)[0].shape
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/mdp/overcooked_env.py", line 239, in featurize_state_mdp
    return self.mdp.featurize_state(state, self.horizon, self.mlam)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/mdp/overcooked_env.py", line 90, in mlam
    force_compute=False)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 809, in from_pickle_or_compute
    return MediumLevelActionManager.compute_mlam(filename, mdp, mlam_params)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 820, in compute_mlam
    mlam = MediumLevelActionManager(mdp, mlam_params=mlam_params)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 780, in __init__
    self.joint_motion_planner = JointMotionPlanner(mdp, mlam_params)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 366, in __init__
    self.all_plans = self._populate_all_plans(debug)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 442, in _populate_all_plans
    if not self.is_valid_jm_start_goal_pair(joint_start_state, joint_goal_state):
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 458, in is_valid_jm_start_goal_pair
    return all([check_valid_fn(joint_start_state[i], joint_goal_state[i]) for i in range(2)])
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 458, in <listcomp>
    return all([check_valid_fn(joint_start_state[i], joint_goal_state[i]) for i in range(2)])
IndexError: tuple index out of range

And I believe there shouldn't be any .pkl files used in order to do this step?

[leobxpan]

Also, if I'm only interested in single-agent environments right now, I guess I could just hack into the featurize_state() function and only return one player state (as well as change all in Overcooked that return two-player states to one) without any other modifications?

[mesutyang97]

No worries, and thanks for working on this.

I tried your method, but it gave out the following error:

>>> env = env.custom_init(base_env, base_env.featurize_state_mdp)
Computing MediumLevelActionManager
Recomputing planner due to: [Errno 2] No such file or directory: '/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/data/planners/cramped_room_single_am.pkl'
Computing MediumLevelActionManager to be saved in /data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/data/planners/cramped_room_single_am.pkl
Traceback (most recent call last):
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 801, in from_pickle_or_compute
    mlam = MediumLevelActionManager.from_file(filename)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 789, in from_file
    return load_saved_action_manager(filename)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/data/planners/__init__.py", line 6, in load_saved_action_manager
    with open(os.path.join(PLANNERS_DIR, filename), 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/data/planners/cramped_room_single_am.pkl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/mdp/overcooked_env.py", line 557, in custom_init
    self.observation_space = self._setup_observation_space()
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/mdp/overcooked_env.py", line 565, in _setup_observation_space
    obs_shape = self.featurize_fn(dummy_state)[0].shape
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/mdp/overcooked_env.py", line 239, in featurize_state_mdp
    return self.mdp.featurize_state(state, self.horizon, self.mlam)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/mdp/overcooked_env.py", line 90, in mlam
    force_compute=False)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 809, in from_pickle_or_compute
    return MediumLevelActionManager.compute_mlam(filename, mdp, mlam_params)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 820, in compute_mlam
    mlam = MediumLevelActionManager(mdp, mlam_params=mlam_params)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 780, in __init__
    self.joint_motion_planner = JointMotionPlanner(mdp, mlam_params)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 366, in __init__
    self.all_plans = self._populate_all_plans(debug)
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 442, in _populate_all_plans
    if not self.is_valid_jm_start_goal_pair(joint_start_state, joint_goal_state):
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 458, in is_valid_jm_start_goal_pair
    return all([check_valid_fn(joint_start_state[i], joint_goal_state[i]) for i in range(2)])
  File "/data/plan2explore/Plan2Explore/lib/python3.6/site-packages/overcooked_ai_py/planning/planners.py", line 458, in <listcomp>
    return all([check_valid_fn(joint_start_state[i], joint_goal_state[i]) for i in range(2)])
IndexError: tuple index out of range

And I believe there shouldn't be any .pkl files used in order to do this step?

Did you pull the newest master? Also, are you using a conda environment as intended for human_aware_rl? If the pkl file isn't there, they should compute it on the spot

[mesutyang97]

Also, if I'm only interested in single-agent environments right now, I guess I could just hack into the featurize_state() function and only return one player state (as well as change all in Overcooked that return two-player states to one) without any other modifications?

Feel free to make modifications, if you are using the environment for single-player setting!

The framework is written with the intention to be used in 2-player setting.

[leobxpan]

Thanks @mesutyang97. The previous build was from pip install. I just built from source and it did compute the .pkl files.

However, when doing custom_init, it gives out the following error:

>>> env = env.custom_init(base_env, base_env.featurize_state_mdp)
Computing MediumLevelActionManager
Recomputing planner due to: [Errno 2] No such file or directory: '/data/overcooked_ai/src/overcooked_ai_py/data/planners/cramped_room_am.pkl'
Computing MediumLevelActionManager to be saved in /data/overcooked_ai/src/overcooked_ai_py/data/planners/cramped_room_am.pkl
Recomputing motion planner due to: [Errno 2] No such file or directory: '/data/overcooked_ai/src/overcooked_ai_py/data/planners/cramped_room_mp.pkl'
Computing MotionPlanner to be saved in /data/overcooked_ai/src/overcooked_ai_py/data/planners/cramped_room_mp.pkl
It took 0.04338574409484863 seconds to create mp
It took 0.306988000869751 seconds to create mlam
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/overcooked_ai/src/overcooked_ai_py/mdp/overcooked_env.py", line 560, in custom_init
    self.observation_space = self._setup_observation_space()
  File "/data/overcooked_ai/src/overcooked_ai_py/mdp/overcooked_env.py", line 569, in _setup_observation_space
    high = np.ones(obs_shape) * max(dummy_mdp.soup_cooking_time, dummy_mdp.num_items_for_soup, 5)
AttributeError: 'OvercookedGridworld' object has no attribute 'soup_cooking_time'

I guess yes, the API of the Overcooked class might be outdated.

[mesutyang97]

OK. This is an oversight on our end. It was updated in rllib.py, but not here. Please change the line to

high = np.ones(obs_shape) * float("inf")

This should resolve it

[mesutyang97]

Thanks again for letting us know about these issues! We are here to support, and your feedback will be incorporated in the next PR

[leobxpan]

My pleasure :) and thanks for your support.

This did solve the issue, and I can now create an environment without any problem. I'm wondering what's the easiest way to make Overcooked compatible with single agent? Seems I'll need to modify a number of functions if I go the hacky way.

[mesutyang97]

That is great news!

In my very humble opinion, it might be the easiest to just have 2 agents, but one of them is a StayAgent which will be located in a insignificant location on the map. You can modify the layout files to achieve this (putting the StayAgent at a place that you won't need).

Hope this helps

[leobxpan]

Thanks @mesutyang97. I think my question is, since I'm working with a 3rd party repo that implements the agent side, if I do not want to hack into their library to put a second StayAgent, I guess I could modify the action space of a two-player environment where the second agent always takes the STAY action. However, I found that the action space in two-player environments is still Discrete(6), i.e., the action space of a single agent. So I'm thinking besides the two intrusive solutions where I can hack into the agent repo or modify the API of your Overcooked to work with single-agent environment, is there an easier way?

[mesutyang97]

Hi @leobxpan . Sorry about the late reply. I was looking for solutions but since I didn't implement the Overcooked's gym API, so I wasn't able to find anything that could help resolve the situation. The author of that part of repo should contact you soon.

[leobxpan]

Thanks @mesutyang97. I already figured out a way to create single-agent environments by writing some wrappers to deal with the single-agent observation space. Thanks anyways.

[micahcarroll]

Let us know if you run into any other issues we can help you with!

[akbir]

OK. This is an oversight on our end. It was updated in rllib.py, but not here. Please change the line to

high = np.ones(obs_shape) * float("inf")

This should resolve it

Hi was this change ever pushed into the repo?

Still seems to still be incorrect on https://github.com/HumanCompatibleAI/overcooked_ai/blob/7e774a1aa29c28b7b69dc0a8903822ac2c6b4f23/src/overcooked_ai_py/mdp/overcooked_env.py#L549

[micahcarroll]

I don't think it was – just updated it directly on master

[richielo]

Hello @micahcarroll , what is the proper way to get a gym environment instance now? I have tried the suggested method above:

mdp = OvercookedGridworld.from_layout_name('asymmetric_advantages') base_env = OvercookedEnv.from_mdp(mdp, horizon = 500) env = gym.make('Overcooked-v0') env = env.custom_init(base_env, base_env.featurize_state_mdp)

and I get this error:

File "C:\Users\richi\Anaconda3\envs\python39\lib\site-packages\overcooked_ai_py\mdp\overcooked_mdp.py", line 1415, in get_counter_objects_dict for obj in state.objects.values(): AttributeError: 'OvercookedGridworld' object has no attribute 'objects'

Any help is greatly appreciated

[micahcarroll]

It seems like get_counter_objects_dict is being passed in an OvercookedGridworld instead of an OvercookedState. @jyan1999 will take a look after his exams next week (or I can look at it later next week), but you're free to try to debug the issue yourself in the meantime and submit a PR!

[richielo]

Hello @micahcarroll , so I have updated the way of initializing the environment to this:

mdp = OvercookedGridworld.from_layout_name('asymmetric_advantages') base_env = OvercookedEnv.from_mdp(mdp, horizon = 500) config_dict = {'base_env' : base_env, 'featurize_fn' : base_env.featurize_state_mdp} env = gym.make('Overcooked-v0', **config_dict)

env = env.custom_init(base_env, base_env.featurize_state_mdp)

print(env.observation_space.low.shape)

But the observation space is simply a Box with shape (96,). Shouldn't it be two Boxes as there are two agents? Please let me know if I am doing anything wrong. I am encountering errors as I add gym wrappers to the environment

[jyan1999]

Hi @richielo, I think I see what you mean and yeah that is probably an oversight on my end (Actually now after looking through it again there are quite a few oversights). Sorry I didn't have a lot of time to look at this earlier this week. I am working on an update to get the observation_space and other things fixed. In the meantime, can you also provide the errors you are getting when adding gym wrappers & how you are planning to use this environment? Thanks!

As clarified by Micah below, the gym environment is not and is not going to be actively maintained. It is definitely possible to adapt the environment to make it support multiagents, though there is no standard interface for that in Gym. To add support multiagent support to this environment, you can start by changing the action and observation space to gym.spaces.Dict object, and add any additionally functionalities you need from there.

[micahcarroll]

Re: "shouldn't there be two Boxes because there are two agents", I don't think the gym API is made to deal with multiple agents, so I believe it's underdetermined in the API whether there should be one or two Boxes in this case.

If this is standard practice, we're happy to update it (it would be useful to see this documented somewhere). Otherwise, I'd stick with having one Box, which implicitly assumes that the agents' observations spaces are all equivalent.

Generally, as we mention in the docstring for Overcooked(gym.Env), this environment will only be somewhat compatible with the gym API, because gym was not designed with multi-agent settings in mind and can't cleanly support them (afaik). Not sure if people have found good workarounds to this in the past. My takeaway has always been that for good multi-agent handling, one should use rllib or other frameworks which have their own multi-agent env API.

In light of that, it doesn't surprise me that some gym wrappers might break when applied directly.

[richielo]

Hello @micahcarroll and @jyan1999, sorry for the delayed response. At least with the multi-agent environments that I have worked with, the observation space is typically a gym.spaces.Tuple or some gym.spaces.Dict and each item would the observation space of an agent. But as you said, there isn't a standard.

My intention is to simply run this environment over MARL algorithms implemented in PyTorch, not going through libraries like stablebaselines or RLLib. But I understand that the environment is not meant to support gym cleanly. I will see if I can find a workaround. Thank you for your help.

[micahcarroll]

Hi @richielo, after looking into this more I've realized we don't actually even use the gym environment anywhere in our codebase (we just decided to keep it from an older version of the code for reference). You're welcome to change the class and submit a PR if you're interested in making it more conformant to normal multi-agent environments that use gym.

[richielo]

@micahcarroll I see. I understand, thank you for the help

[micahcarroll]

Of course! 😄 Let us know if you have any further questions with the repo!