search

JordanAsh / badge

An implementation of the BADGE batch active learning algorithm.
193 stars 32 forks source link

Got different accuracy results #1

Open Sungwon-Han opened 4 years ago

Sungwon-Han commented 4 years ago

Thank you for sharing the source code for this great work.

I am trying to replicate some of your baseline results. However, the result from the baseline (LeastConfidence) I got is much better than the one reported in your paper. Here is my result accuracy plot (It achieves 40% test accuracy much earlier than the reported accuracy in the paper).

Query size: 100 Dataset: CIFAR-10 Network: Resnet Strategy: LeastConfidence image

Should I do something to replicate your results?

JordanAsh commented 4 years ago

Yes, the default learning rate isn't correct. It should be an order of magnitude lower, I believe.

On Wed, Jul 29, 2020 at 7:21 PM Sungwon Han notifications@github.com wrote:

Thank you for sharing the source code for this great work.

I am trying to replicate some of your baseline results. However, the result from the baseline (LeastConfidence) I got is much better than the one reported in your paper. Here is my result accuracy plot (It achieves 40% test accuracy much earlier than the reported accuracy in the paper).

Query size: 100 Dataset: CIFAR-10 Network: Resnet Strategy: LeastConfidence [image: image] https://user-images.githubusercontent.com/17738624/88863172-25f6f100-d23d-11ea-9933-c0848eb44727.png

Should I do something to replicate your results?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/JordanAsh/badge/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEAIDG6QH7DZA46YMCWZVFLR6CVFXANCNFSM4PMI6ZDA .

Sungwon-Han commented 4 years ago

Even after changing the learning rate as same as reported in the paper, I can't replicate the paper's result. Are there any other things that I should do for replication?

JordanAsh commented 4 years ago

Right, I'm saying that I believe the learning rate in the paper is off by an order of magnitude.

On Wed, Jul 29, 2020 at 10:56 PM Sungwon Han notifications@github.com wrote:

Even after changing the learning rate as same as reported in the paper, I can't replicate the paper's result. Are there any other things that I should do for replication?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/JordanAsh/badge/issues/1#issuecomment-666058643, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEAIDG7TY6TI6CNYMIEK2QDR6DOO7ANCNFSM4PMI6ZDA .

Sungwon-Han commented 4 years ago

Then, can you provide a command to replicate the paper's result? Or, are there any seeds to replicate the paper's result?