Few tips about how to run the code

[CottePutman]

Introduction

I successfully run the code on Ubuntu 20.04 within the following environment:

• Python == 3.7.2
• baselines == 0.1.6 (local)
• Cython == 0.29.37
• gym == 0.10.8
• mujoco-py == 2.0.2.8
• multiagent == 0.0.1 (local)
• seaborn == 0.12.2
• torch == 1.12.1+cu116

Problems during the installation mostly occured on mujoco-py and to be honest there's just no way around it. Few tips I can offer is to stay patient and it will somehow work out :)

When running the code, thing gets stricker. I tried three env_id and only one managed to "output results". Here are the problems I came across and my solutions to it:

Datasets

1. simple_tag / simple_world (didn't run out)

- In algorithms/maddpg,py/init_from_env(), line 453:

num_in_pol = obsp.shape[0]    #observation space
num_out_pol = acsp.shaep[0]   #action space

However, ascp doesn't possess the shape[0] attribute, which leads to IndexOutBounday exception (Also, code follows it comes up with access to acsp.high[0] which would cause NoAttribute exception).

I failed to find out the possible reasons by looking following the stack up to the multiagent-particle-envs/multiagent/environment.py/__init__(), where the env is created. So I guess it either involves version incompatibility of multiagent (but it only comes with one version 0.0.1 and that is the one installed), or the code might be modified to satisfy other datasets but those two.

- In utils/make_env.py/make_env():

if benchmark or scenario_name == 'simple_spread':
    env = MultiAgentEnv(world, scenario.reset_world, scenario.reward, scenario.observation, scenario.benchmark_data, discrete_action=discrete_action)
else:
    env = MultiAgentEnv(world, scenario.reset_world, scenario.reward, scenario.observation, discrete_action=discrete_action)

MultiAgentEnv() doesn't have parameter discrete_action, the implement of it only has discrete_action_space and discrete_action_input, which is set True and False by default seperately. I have no idea but to simply delete the parameter and everything seems fine to me.

2. HalfCheetah v2

I noticed that there are a few if else desinated to seperate HalfCheetah v2 apart from others, so I decided to give it a try. With the two modifcations above, I came across the following problem:

In algorithms/maddpg,py/update(), line 257:

dist = torch.distributions.Normal(self.omar_mu, self.omar_sigma)

This line will throw an exception about ValueError: Expected parameter scale of distribution Normal to satisfy the constraint GreaterThan..., which means this function can only take values that are larger than 0 as input but is called that way.

The reference of torch.distributions.Normal() is put within a for recycle and even I set self.omar_mu and self.omar_sigma to meet the requirement, it will throw the same exception right on the second recycle. Therefore, I tried torch.clamp() and it seems worked. Here is the modified code concerned:

if self.is_mamujoco:
    ...
    # Adding a tiny postive shift
    self.omar_mu += 1e-6
    self.omar_sigma += 1e-6      

    for iter_idx in range(self.omar_iters):
        dist = torch.distributions.Normal(self.omar_mu, self.omar_sigma) 
        ...
        updated_mu = torch.mean(elite_acs, dim=1)
        updated_sigma = torch.std(elite_acs, dim=1)

        # Make sure the updated parameters are bigger than 0
        updated_mu = torch.clamp(updated_mu, min=1e-6)
        updated_sigma = torch.clamp(updated_sigma, min=1e-6)
        ...

Notice: I have no idea how this could affect the code so be cautious when trying

Then the code was successfully running. Since there is no results outputing function I can only monitor the process in debug mode, eventually reached an eval_return of around 1800 in around 370k steps.

Hope this could help!

[CottePutman]

Update

In the maddpg-pytorch README file I found that this code requires a specific fork of multiagent-particle-envs. By applying so the problem in utils/make_env.py/make_env() can be fixed. Also, ascp.shape[0] is fixed.