Performance

[Ferenas]

Hi, thanks for your work. I have run your code in Yahoo-dataset following the basic settings, however, I can only get 56% accuracy. I was wondering why the performance cannot reach the best recorded in the paper

[mikigom]

Thank you for your interest.

Unfortunately, the current benchmark of partial label learning is highly dependent on the random seed of the train/test split due to the dataset size or the nature of the problem. Moreover, it is even more serious because most of the existing papers do not specifically disclose how the train/test split is divided. This is why we repeated 10-fold validation 5 times and reported the averaged performance of 50 runs, respectively.

Therefore, in benchmarking, rather than observing a single performance, I hope that you judge it by the distribution of iterative performance. If you take the average of ~50 runs in the same way as the benchmark in this paper, you will get a performance metric that is close to the value reported in our paper.

[Ferenas]

Thanks for your response！In fact，i was wondering what the difference between your method and with the CC method，Provably Consistent Partial-Label Learning

发自我的iPhone

在 2021年7月15日，下午12:56，mikigom @.***> 写道：

 Thank you for your interest.

Unfortunately, the current benchmark of partial label learning is highly dependent on the random seed of the train/test split due to the dataset size or the nature of the problem. Moreover, it is even more serious because most of the existing papers do not specifically disclose how the train/test split is divided. This is why we repeated 10-fold validation 5 times and reported the performance of 50 runs, respectively.

Therefore, in benchmarking, rather than observing a single performance, I hope that you judge it by the distribution of iterative performance. If you take the average of ~50 runs in the same way as the benchmark in this paper, you will get a performance metric that is close to the value reported in our paper.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[mikigom]

Short Answer

No difference in implementation. Only the constant term of loss is different.

Long Answer

As described in the paper, our goal in this study is just to demonstrate the effectiveness of the use of the existing naive loss for training neural networks to solve PLL. According to the paper "Provably Consistent Partial-Label Learning", devising CC-risk method was motivated by the works on recent multiple complementary label probem. However, from our point of view, CC-risk is equivalent to naive loss. So, we introduced the theoretical analysis introduced in the CC risk paper along with EPRM Learnability as a generalization bound on naive loss.