search

JordanAsh / badge

An implementation of the BADGE batch active learning algorithm.
197 stars 32 forks source link

Different results #4

Open hieuvt29 opened 3 years ago

hieuvt29 commented 3 years ago

Hi authors, It's great that you publish the source code, but the default hyper-parameters seem not to be correctly tuned, so could you please share the hyper-params to produce the graph below (taken from your paper) I also add the graph showing the results that I ran with hyper-params from your source code with command python run.py --model vgg --nQuery 100 --data CIFAR10 --alg badge

Reported in the paper: image Run default code: (the x-axis is the number of data point, divided by 100 - the query batch size) The gap is nearly 10% even with more data image

JordanAsh commented 3 years ago

Hello,

The default learning rate in our implementation is different from what we use in our paper. To me it looks like it's probably too low -- I'd try increasing it by an order of magnitude.

Thanks, Jordan

On Wed, Jan 6, 2021 at 10:56 AM Vu Trong Hieu notifications@github.com wrote:

Hi authors, It's great that you publish the source code, but the default hyper-parameters seem not to be correctly tuned, so could you please share the hyper-params to produce the graph below (taken from your paper) I also add the graph showing the results that I ran with hyper-params from your source code with command python run.py --model vgg --nQuery 100 --data CIFAR10 --alg badge

Reported in the paper: [image: image] https://user-images.githubusercontent.com/18468247/103787805-a8979e00-5070-11eb-8752-aa00abfa6f89.png Run default code: (the x-axis is the number of data point, divided by 100

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/JordanAsh/badge/issues/4, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEAIDG7TJN6CU26HDCUDNKLSYSB4VANCNFSM4VXXAY6A .

hieuvt29 commented 3 years ago

Thank you for replying, I assume you're taking about the learning rate for training the task model, but as I understand based on the code here https://github.com/JordanAsh/badge/blob/master/query_strategies/strategy.py#L63, the model will be trained until getting 99% accuracy on training set, so it'll fit the training data without really care about the learning rate (I also checked the log). Should I change something else?

JordanAsh commented 3 years ago

I'm not sure what you mean by task model. Even though different learning rates may all achieve perfect training accuracy, they often produce different test accuracies.

On Wed, Jan 6, 2021 at 10:53 PM Vu Trong Hieu notifications@github.com wrote:

Thank you for replying, I assume you're taking about the learning rate for training the task model, but as I understand based on the code here https://github.com/JordanAsh/badge/blob/master/query_strategies/strategy.py#L63, the model will be trained until getting 99% accuracy on training set, so it'll fit the training data without really care about the learning rate (I also checked the log). Should I change something else?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/JordanAsh/badge/issues/4#issuecomment-755866389, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEAIDG55YLAE6265KJGVSBDSYUV2LANCNFSM4VXXAY6A .

hieuvt29 commented 3 years ago

"task model" here I mean the classifier (or the learner), I agree that different learning rates will eventually get different accuracies on test set but the variance should not be that big (10%). Anyway, thank you for your replies!