jakegrigsby
commented
1 week ago Hi @Lan131, the wandb link i put in the other issue (#62) has a full history of the command already done for you (on an A5000).
https://wandb.ai/jakegrigsby/amago-v3-reference/runs/gq9s8vxs?nw=nwuserjakegrigsby
The default x-axis of wandb "Step" doesn't mean anything useful --- it's the number of times wandb.log has been called. Instead, you want to display results by Wall Time, total_frames, Epoch, or the total frames of any particular task name (just search "total_frames" and you'll see 45 extra options).
The original meta-world RL^2 results are listed at 400M total timesteps (summing over the 45 tasks). The result in our paper Figure 4 goes for 100M. The wandb above happens to go to 200M before I stopped it. If you're looking for 1M total timesteps with an RL^2-like algo... I wouldn't recommend that... but I assume your paper is specifically about sample efficiency. It would take a totally different set of hparams to give that a reasonable shot.
The data collection process per epoch is described in a new tutorial here.
The parallel actors (40 in the command you're using), are sampling randomly from the 45 tasks each meta-reset. The ratio of experience collected to learning updates is the main factor in sample efficiency. You'd probably want to use fewer actors and do more gradient steps. Also, you'd probably want to use a shorter policy sequence length than the 256 used in the command. The "meta" aspect of MetaWorld ML45 actually doesn't take much memory and shorter policy lengths are generally more sample efficient. Something like 32 or even 16 might learn faster.
Episodes last for 500 timesteps, which is based on the garage max_pax_length. Some papers change it and I just pushed a commit that lets you change it if you want. Shorter would probably be better for sample efficiency because the way success rates are computed in Meta-World means that the agent is just kinda sitting there wasting steps once it's finished with most of the tasks.
Lan131
I was wondering how fast this is supposed to run? Over 6 hours it only trained 2k steps according to the wandb output and I have an A6000.
I want to run it for 1 million env steps. I believe that in metaworld max time steps per episode is 150. What does the parallel_actors argument doing? Maybe thats slowing it.
python ./examples/07_metaworld.py --run_name metaworld_ml45 --benchmark ml45 --buffer_dir buffer --parallel_actors 40 --memory_size 320 --timesteps_per_epoch 150 --agent_type multitask --max_seq_len 256 --memory_layers 3 --dset_max_size 25000 --epochs 6667 --val_interval 40