cannot reproduce result in ARS paper

[fiberleif]

Dear authors, just for halfcheetah-v1, l use multiple seeds to try to get the exp result in table1&table2 in paper: python code/ars.py but it seems negative. could you give some guide? thanks so much!

[hmania]

Hi,

I would be happy to help. Can you describe the steps you are taking? Please DM me.

Best, Horia

[fiberleif]

Thanks! l follow the hyperparameter of Table 9. e.g. for swimmer-v1, nd=du=1 seems not stable for improvement, for walker2d-v1, use the nd=40, du=30, l can't also achieve the average curve showed in Figure 2. does it relate to export MKL_NUM_THREADS=1? l don't set that before experiment l remember.

[hmania]

In the paper we point that the evaluation of ARS on the MuJoCo locomotion tasks has high variance. In particular, Figure 3 shows the performance of ARS when evaluated over 100 random seeds. Figure 3 clearly shows that the performance degrades when the random seed is varied. In other words, to obtain performance comparable to that shown in Table 1/ Figure 2 you need re-tune ARS or you need to have the three random seeds we used (237, 759, 959)

[fiberleif]

Thanks so much for such detailed reply~ and the evaluation over 100 random seeds is objective and realizable.

Thanks!