search

Innixma / autogluon-benchmarking

Benchmarking Utilities for AutoGluon
Apache License 2.0
20 stars 5 forks source link

Kaggle benchmarks #4

Open mglowacki100 opened 3 years ago

mglowacki100 commented 3 years ago

Hi Nick

I've read yours https://arxiv.org/abs/2003.06505 I think that mentioning that for at least couple of kaggle competitions obtaining top-scores is not possible without using data-leaks would be more fair to automated ml and would add some value to paper. For example:

  1. https://www.kaggle.com/c/santander-customer-transaction-prediction: https://www.kaggle.com/c/santander-customer-transaction-prediction/discussion/85125
  2. https://www.kaggle.com/c/santander-value-prediction-challenge: https://www.kaggle.com/c/santander-value-prediction-challenge/discussion/63907
  3. https://www.kaggle.com/c/ieee-fraud-detection: https://www.kaggle.com/c/ieee-fraud-detection/discussion/111284 Detection and exploit such leaks is hard even for human.

Btw. do you have plans for paper release with updated benchmarks?

Anyways great work :)

Innixma commented 3 years ago

Thanks for the info! We will try to incorporate it into the next version of the paper.

Regarding updated benchmarks, we are planning to have them sometime in March/April 2021 as we are currently working on v0.1 release.

[Update 2022]: This timeline was shifted and we are planning to have the new paper release in March/April 2022 (Accepted at JMLR, preparing camera ready).