Maml implementation

[heraldiclily]

hello @llan-ml

Thank you for sharing this Maml implementation on ray. Have you been able to fix the raygeterror issue with ray support team? I will be gratefull if you could enrich this page with some implementation instructions.

[llan-ml]

I am using https://github.com/ray-project/ray/commit/20c7fad4f4426582da89765c607c487c875bc7ab now, and it works well without RayGetError. But the RLlib codes of this commit are not compatible with this codebase, and it should be easy to fix if you are familiar with RLlib codes (most cases are that APIs add new arguments).

As for Carla, I do not have Carla installed, and thus I cannot test with the Carla example. If you encounter any errors/bugs, feel free to ask.

[smiler80]

hello @llan-ml,

Thank you for the shared code. Is it specific to some environments or generic? I would like to apply maml to gym env but the available codes are generally customized to certain application (halfCheetah, mountain car...). I went through your code and found no environment name, to which task did you apply it successfully?

[llan-ml]

hi @bbacem80 The code is developed based on Ray and RLlib.

We need to define a env_creator, register it with Ray, https://github.com/llan-ml/maml-on-ray/blob/6e58b1a65fca4b278d26f994b139fa3b233a91ac/main.py#L21-L22

The run_experiments will automatically pass the env name to the __init__ of the agent, and then the env name is used to construct PolicyEvaluator.

In principle, the MAML code is suitable to any environments as long as the environment conforms to RLlib design.

[smiler80]

Thank you @llan-ml again for your explanations.

Since I'm beginner in RLLIB toolkit, and while I'm deepening my knowledge on the subject, I would be grateful if you can provide initiating instructions on how to run your code maml with a Gym environment, i.e. after the git clone of ray-project:

• where to git clone maml-on-ray files (same working folder as ray-project)
• how to call an existing environment (as an argument or within the code)
• which RL algorithm is used (A3C, PPO)

[llan-ml]

hi @bbacem80

• You can first install ray via pip install ray==0.6.1, and then just clone maml-on-ray into wherever you want. Run main.py can further take advantages of Ray Tune. Or, you can just run maml.py.
• Due to that MAML does inner/fast update for each task, we need to independently sample episodes for each task, which means that we should be able to assign a specific desired task to an environment, like https://github.com/llan-ml/maml-on-ray/blob/6e58b1a65fca4b278d26f994b139fa3b233a91ac/point_env.py#L34 However, Gym does not define a reset_args for the method env.reset. Therefore, we cannot directly use Gym environments for MAML training. Instead, you can use PointEnv for testing, as in the original paper. Another thing is that since we will use the env class for distributed sampling (or remote/different processes), we need to register it with Ray first. Here, I call PointEnv via https://github.com/llan-ml/maml-on-ray/blob/6e58b1a65fca4b278d26f994b139fa3b233a91ac/main.py#L20-L22 and https://github.com/llan-ml/maml-on-ray/blob/6e58b1a65fca4b278d26f994b139fa3b233a91ac/main.py#L43-L51 Or, https://github.com/llan-ml/maml-on-ray/blob/6e58b1a65fca4b278d26f994b139fa3b233a91ac/maml.py#L171-L180 and https://github.com/llan-ml/maml-on-ray/blob/6e58b1a65fca4b278d26f994b139fa3b233a91ac/maml.py#L196
• I just name the inner update as A3C, and the outer update as PPO.

The code in this repo was just used for debugging a ray bug before, and is obsolete now since the interfaces of RLlib have changed. Definitely, directing running the code will raise some errors (most cases are that new APIs use some new arguments).

Later, I will clean up my local codebase and upload a new version, but you can try this now.