search

cbfinn / maml_rl

Code for RL experiments in "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks"
Other
624 stars 180 forks source link

Installation/replicating point mass experiments #3

Open ajdroid opened 6 years ago

ajdroid commented 6 years ago

Hi, I've been trying to replicate some of your results and compare against Reptile, for the reinforcement learning case. Specifically, I'm trying to run the point mass experiments.

After installing everything and sourcing the conda env rllab3 and running python maml_examples/maml_trpo_point.py, it complains about Theano and Lasagne not being installed.

From the README, it seems like this was built on the Tensorflow version of rllab. Does it still need Theano or am I doing something wrong?

PS: Thanks for open-sourcing your work!

cbfinn commented 6 years ago

Yes, unfortunately, the Tensorflow version of rllab does require theano.

FYI, with regard to reptile, the easiest thing to try would be first-order MAML by adding a stop gradient to the existing code. I believe that I tried this awhile back, and found that it didn't work. But, I can't remember exactly which setting I was looking at.

ajdroid commented 6 years ago

Alright, so I was able to get the train script to run with theano 0.7 (nothing newer works). (Worth adding in the README, maybe? I'm trying to add more instructions to everything so if I understand everything properly, I can send a PR when done if you'd like)

Also, when I run python maml_examples/trpo_point.py which I assume is the baseline, it throws:

AttributeError: 'GaussianMLPPolicy' object has no attribute 'all_param_vals'

To test (after training with maml_examples/maml_trpo_point.py) I'm trying to use the script maml_examples/test_maml_point.py. It however imports the Mujoco Walker2D env, which I don't have access to currently. It doesn't seem to be used inside that file. Is it required?

At this point, my question also is, what is correct order of scripts to run to replicate your 2D pointmass experiments from the paper? I was trying:

  1. run python maml_examples/maml_trpo_point.py
  2. run python maml_examples/trpo_point.py
  3. run python maml_examples/test_maml_point.py Maybe I'm doing this wrong?
cbfinn commented 6 years ago

The particular version of theano that works is in environment.yml.

It however imports the Mujoco Walker2D env

Sorry about that. It is not needed.

what is correct order of scripts to run to replicate your 2D pointmass experiments from the paper? I was trying...

Yes, that's correct. Step 1 will train MAML and step 2 will train the baseline.

I don't have time to look into the error right now, but I can look into it later this week. I suspect that it is a very simple fix.

ajdroid commented 6 years ago

Okay, thanks! :)

Meanwhile, I ran the testing code for just the MAML training output. This assertion always returns false. The last column of the csv is, in fact "AveragePolicyStd" rather than "AverageReturn".

I've changed it to find "Average Return" now, but does this unexpected behaviour mean something else is wrong in the pipeline?

ajdroid commented 6 years ago

@cbfinn Hi, sorry did you have any time to look at the bug?

shubhamag commented 6 years ago

Also, when I run python maml_examples/trpo_point.py which I assume is the baseline, it throws:

AttributeError: 'GaussianMLPPolicy' object has no attribute 'all_param_vals'

Hi, I'm encountering the same issue as well. @ajdroid @cbfinn Could either of you figure out a fix for this ?

ajdroid commented 6 years ago

Hi, just asking again if you had any time to look at this @cbfinn !

zackenton commented 6 years ago

I ran into this too, I think this can be fixed by changing the -1 to 1 in lines 102 and 104 https://github.com/cbfinn/maml_rl/blob/master/maml_examples/test_maml_point.py#L102. You may also need to change 'w' to 'wb' in https://github.com/cbfinn/maml_rl/blob/master/maml_examples/test_maml_point.py#L120

hl00 commented 6 years ago

default Hi, have any of you fixed this issue? @cbfinn @ajdroid @shubhamag

yahya-sattar commented 5 years ago

@cbfinn any update on the following problem? AttributeError: 'GaussianMLPPolicy' object has no attribute 'all_param_vals'

yahya-sattar commented 5 years ago

Alright, so I was able to get the train script to run with theano 0.7 (nothing newer works). (Worth adding in the README, maybe? I'm trying to add more instructions to everything so if I understand everything properly, I can send a PR when done if you'd like)

Also, when I run python maml_examples/trpo_point.py which I assume is the baseline, it throws:

AttributeError: 'GaussianMLPPolicy' object has no attribute 'all_param_vals'

To test (after training with maml_examples/maml_trpo_point.py) I'm trying to use the script maml_examples/test_maml_point.py. It however imports the Mujoco Walker2D env, which I don't have access to currently. It doesn't seem to be used inside that file. Is it required?

At this point, my question also is, what is correct order of scripts to run to replicate your 2D pointmass experiments from the paper? I was trying:

1. run `python maml_examples/maml_trpo_point.py`

2. run `python maml_examples/trpo_point.py`

3. run `python maml_examples/test_maml_point.py`
   Maybe I'm doing this wrong?

@ajdroid Hi, Were you able to run step 2?

GbengaOdesanmi commented 5 years ago

@all I run python maml_trpo_point.py but am having this error

python3 /home/gbenga/Documents/2019 vrep code/maml_rl-master/scripts/run_experiment_lite.py --n_parallel '1' --snapshot_mode 'last' --seed '1' --plot 'False' --exp_name 'trpomaml1_fbs20_mbs40_flr_0.5metalr_0.01_step11' --log_dir '/home/gbenga/Documents/2019 vrep code/maml_rl-master/data/local/vpg-maml-point100/trpomaml1_fbs20_mbs40_flr_0.5metalr_0.01_step11' --use_cloudpickle 'False' --args_data 'gANjcmxsYWIubWlzYy5pbnN0cnVtZW50ClN0dWJNZXRob2RDYWxsCnEAKYFxAX1xAihYBgAAAF9fYXJnc3EDKGNybGxhYi5taXNjLmluc3RydW1lbnQKU3R1Yk9iamVjdApxBCmBcQV9cQYoWAQAAABhcmdzcQcpWAYAAABrd2FyZ3NxCH1xCShYAwAAAGVudnEKaAQpgXELfXEMKGgHKWgIfXENWAsAAAB3cmFwcGVkX2VudnEOaAQpgXEPfXEQKGgHKWgIfXERaApoBCmBcRJ9cRMoaAcpaAh9cRRYCwAAAHByb3h5X2NsYXNzcRVjbWFtbF9leGFtcGxlcy5wb2ludF9lbnZfcmFuZGdvYWwKUG9pbnRFbnZSYW5kR29hbApxFnVic2gVY3JsbGFiLmVudnMubm9ybWFsaXplZF9lbnYKTm9ybWFsaXplZEVudgpxF3Vic2gVY3NhbmRib3gucm9ja3kudGYuZW52cy5iYXNlClRmRW52CnEYdWJYBgAAAHBvbGljeXEZaAQpgXEafXEbKGgHKWgIfXEcKFgEAAAAbmFtZXEdaBlYCAAAAGVudl9zcGVjcR5jcmxsYWIubWlzYy5pbnN0cnVtZW50ClN0dWJBdHRyCnEfKYFxIH1xIShYBAAAAF9vYmpxImgLWAoAAABfYXR0cl9uYW1lcSNYBAAAAHNwZWNxJHViWA4AAABncmFkX3N0ZXBfc2l6ZXElRz/gAAAAAAAAWBMAAABoaWRkZW5fbm9ubGluZWFyaXR5cSZjdGVuc29yZmxvdy5weXRob24ub3BzLmdlbl9ubl9vcHMKcmVsdQpxJ1gMAAAAaGlkZGVuX3NpemVzcShLZEtkhnEpdWgVY3NhbmRib3gucm9ja3kudGYucG9saWNpZXMubWFtbF9taW5pbWFsX2dhdXNzX21scF9wb2xpY3kKTUFNTEdhdXNzaWFuTUxQUG9saWN5CnEqdWJYCAAAAGJhc2VsaW5lcStoBCmBcSx9cS0oaAcpaAh9cS5oHmgfKYFxL31xMChoImgLaCNoJHVic2gVY3JsbGFiLmJhc2VsaW5lcy5saW5lYXJfZmVhdHVyZV9iYXNlbGluZQpMaW5lYXJGZWF0dXJlQmFzZWxpbmUKcTF1YlgKAAAAYmF0Y2hfc2l6ZXEySxRYDwAAAG1heF9wYXRoX2xlbmd0aHEzS2RYDwAAAG1ldGFfYmF0Y2hfc2l6ZXE0SyhYEAAAAG51bV9ncmFkX3VwZGF0ZXNxNUsBWAUAAABuX2l0cnE2S2RYCAAAAHVzZV9tYW1scTeIWAkAAABzdGVwX3NpemVxOEc/hHrhR64Ue1gEAAAAcGxvdHE5iHVoFWNzYW5kYm94LnJvY2t5LnRmLmFsZ29zLm1hbWxfdHJwbwpNQU1MVFJQTwpxOnViWAUAAAB0cmFpbnE7KX1xPHRxPVgIAAAAX19rd2FyZ3NxPn1xP3ViLg==' python3: can't open file '/home/gbenga/Documents/2019': [Errno 2] No such file or directory

GUO99131 commented 2 years ago

same problem. Have u fixed? @GbengaOdesanmi