search

distillpub / post--bayesian-optimization

Exploring Bayesian Optimization
https://distill.pub/2020/bayesian-optimization/
72 stars 16 forks source link

Moons dataset looks weird #16

Open kennysong opened 4 years ago

kennysong commented 4 years ago

Nice article! The image of the moons dataset at the end of the article looks a bit strange since the points are heavily overlapping and not in a moon shape.

Screen Shot 2020-05-06 at 1 35 34 PM

Is it the correct image?

apoorvagnihotri commented 4 years ago

Thanks a lot for the appreciation. :)

Yes, it is the correct image, though we have used a variance of 1 while creating the dataset. You can find the code used for replicating the above image here.

kennysong commented 4 years ago

Thanks for the reference! Was the reason for the high noise just to make the task harder and spread out the performance of different hyperparameter sets?

apoorvagnihotri commented 4 years ago

Hi @kennysong. To be honest, we didn't consider varying the noise. But I think your idea should be valid in this case.

I did some experiments with low noise and Bayesian Optimization (BO) is performing better. PFA, moon's data results on SVM.

New, simple moon’s dataset.

New accuracies on using an SVM.

Old accuracy on using an SVN.

New accuracies on using an RF.

Old accuracies on using an RF.

Our task is much difficult--because of using a very high noise in dataset--in the original article. Therefore, it was difficult for the BO to perform. As seen here, reducing the complexity helped BO achieve better results.

You can have a look at the ipynb used here.

Thanks a lot for pointing this out. I’ll keep this issue open as I think this is an important point. :)

kennysong commented 4 years ago

Oh cool! It does look like the original harder task spread out the performance of different algorithms more. With less noise, all approaches seem to do better and converge to the global optimum.

Thanks for following up!