Cannot reproduce the results of Ackley-20d in the paper

facebookresearch / LaMCTS

The release codes of LA-MCTS with its application to Neural Architecture Search.

Other

463 stars 71 forks source link

Cannot reproduce the results of Ackley-20d in the paper #20

Closed gloaming2dawn closed 2 years ago

gloaming2dawn commented 3 years ago

Hi, thank you for your good work.

Now I am trying to reproduce the results of Ackley-20d in the LAMCTS paper (Fig. 4). I use the same hyper-parameters as (cp=1, leaf size=20) and run: cd LA-MCTS python run.py --func ackley --dims 20 --iterations 1000

I have repeated the process for 20 times, however I cannot obtain the the results in the paper. Is there anything I need to change to reproduce the results?

linnanwang commented 3 years ago

What are the results you have got on your side? Sorry I just got noticed about this issue.

gloaming2dawn commented 3 years ago

What are the results you have got on your side? Sorry I just got noticed about this issue.

I found the result of LAMCTS-BO (the base model is BO) is similar to naive BO.

linnanwang commented 3 years ago

Hello,

Can you please provide more data for me to analyze? E.g., what are the results on your side? Are you able to replicate Ackley-10d and see the difference? Thanks.

gloaming2dawn commented 3 years ago

Hi, thank you for your reply. Just see the image below. All I did is to Open file LA-MCTS/run.py, change line 57 to leaf_size =20. Then run the following command: python run.py --func ackley --dims 20 --iterations 1000

gloaming2dawn commented 3 years ago

As it is easy to repeat the process, could you please try to reproduce the results of Ackley-20d again using LAMCTS-BO from the current source code? I think there may be something wrong in the current code.

linnanwang commented 3 years ago

Thanks, Likely this is due to the configurations. I will look into this and get back to you later (after Nov-16th ;), a deadline for me).

gloaming2dawn commented 2 years ago

Hi, I have found why it failed. Just modify line 243 in MCTS.py, leaf.propose_samples_bo( 1, path, self.lb, self.ub, self.samples ) to leaf.propose_samples_bo( 10, path, self.lb, self.ub, self.samples )

linnanwang commented 2 years ago

Hello,

I'm glad you figure it out. Can you leave your email? I will send you the codes to Ackley-20d for reproduction.

gloaming2dawn commented 2 years ago

Thank you. My email is jiayuzhao199567@163.com. Could you please also provide the codes of Half-Cheetah function and And function?

linnanwang commented 2 years ago

sent, the files also contain the original log.

linnanwang commented 2 years ago

Hey

Did you receive my codes? Have you resolved this issue? Thanks.