search

imirzadeh / Teacher-Assistant-Knowledge-Distillation

Using Teacher Assistants to Improve Knowledge Distillation: https://arxiv.org/pdf/1902.03393.pdf
MIT License
256 stars 47 forks source link

Teacher (resnet26) best accuracy #8

Closed TeerathChandani closed 5 years ago

TeerathChandani commented 5 years ago

Hello Imirzadeh,

Can you tell me the best accuracy of teacher?

If you can share weight file of teacher, it would be better?

Thanks.

imirzadeh commented 5 years ago

The validation accuracy of the teacher is 92.48

Here is the weight file: https://www.dropbox.com/s/bwsj585824hwd39/resnet26_d9cc8dae25984e2b9829d080c9c0e024_best.pth.tar?dl=0

TeerathChandani commented 5 years ago

But I am getting teacher accuracy that is almost 82%.. Why I am getting this low accuracy?

imirzadeh commented 5 years ago

What do you mean? You are using my code to train a resnet 26 and getting 82% ? If yes, what is your setting? how many epochs, ....?

TeerathChandani commented 5 years ago

Yes Epoch : 160 Seed(2)

As other lambda and T are not used. So I just commented them.

Thanks.

TeerathChandani commented 5 years ago

Secondly there was issue in 176 of train.py

I replaced

num_classes = 100 if dataset == 'cifar100' else 'cifar10'

with

num_classes = 100 if dataset == 'cifar100' else 10

imirzadeh commented 5 years ago

Hmm Have you tried with another seed? Maybe it's a bad local optima. If it still has the problem, let me know so I can debug it (hopefully this weekend)

TeerathChandani commented 5 years ago

Yes. I tried like 5, 20, and 50. If we comment this seed, what do you think? I really do not know. why this is happening..

Thanks for your super fast reply. :)

No issue take your time. I am waiting till this weekends

TeerathChandani commented 5 years ago

By the way, which seed value your used for this training(teacher-resnet26)?

imirzadeh commented 5 years ago

Here is the complete execution for resnet26 training: https://www.comet.ml/cifar10/c10res26/d9cc8dae25984e2b9829d080c9c0e024

You can see the training charts in the "Charts" section on the left menu. The code is the dirty version of this repository since I cleaned the code before I put it on Github.

TeerathChandani commented 5 years ago

Thank you so much for your help. Let me try this.

TeerathChandani commented 5 years ago

Hello Dear Iman Mirzadeh,

Thanks for your help. I really got it. Did you train from Teacher(resnet26) to student(resnet8) directly? I need that model, can you help me? I trained, but I am getting very less accuracy almost 80% because may be I am setting random parameters. You have nni lib and you can easily do it.

Again Thanks for providing assistance.

Waiting for your response.

Thanks, Teerath Kumar

On Wed, 7 Aug 2019 at 04:26, Iman Mirzadeh notifications@github.com wrote:

Here is the complete execution for resnet26 training: https://www.comet.ml/cifar10/c10res26/d9cc8dae25984e2b9829d080c9c0e024

You can see the training charts in the "Charts" section on the left menu. The code is the dirty version of this repository since I cleaned the code before I put it on Github.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/imirzadeh/Teacher-Assistant-Knowledge-Distillation/issues/8?email_source=notifications&email_token=AMF5TERUC4PPHP46KVEHKUTQDHF7HA5CNFSM4IJRYKA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3WGPXQ#issuecomment-518809566, or mute the thread https://github.com/notifications/unsubscribe-auth/AMF5TERW3ZQGEUNYYFDH7DDQDHF7HANCNFSM4IJRYKAQ .

TeerathChandani commented 5 years ago

Hello Dear Iman Mirzadeh,

Thanks for your help. I really got it. Did you train from Teacher(resnet26) to student(resnet8) directly? I need that model, can you help me? I trained, but I am getting very less accuracy almost 80% because may be I am setting random parameters. You have nni lib and you can easily do it.

Again Thanks for providing assistance.

And also can you provide weight file of below network for CIFAR100?

  1. Res110
  2. Res32
  3. Res20

It would be really great if you provide.

Thanks, Teerath Kumar

On Tue, 20 Aug 2019 at 10:53, teerath kumar teerathkumar142@gmail.com wrote:

Hello Dear Iman Mirzadeh,

Thanks for your help. I really got it. Did you train from Teacher(resnet26) to student(resnet8) directly? I need that model, can you help me? I trained, but I am getting very less accuracy almost 80% because may be I am setting random parameters. You have nni lib and you can easily do it.

Again Thanks for providing assistance.

Waiting for your response.

Thanks, Teerath Kumar

On Wed, 7 Aug 2019 at 04:26, Iman Mirzadeh notifications@github.com wrote:

Here is the complete execution for resnet26 training: https://www.comet.ml/cifar10/c10res26/d9cc8dae25984e2b9829d080c9c0e024

You can see the training charts in the "Charts" section on the left menu. The code is the dirty version of this repository since I cleaned the code before I put it on Github.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/imirzadeh/Teacher-Assistant-Knowledge-Distillation/issues/8?email_source=notifications&email_token=AMF5TERUC4PPHP46KVEHKUTQDHF7HA5CNFSM4IJRYKA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3WGPXQ#issuecomment-518809566, or mute the thread https://github.com/notifications/unsubscribe-auth/AMF5TERW3ZQGEUNYYFDH7DDQDHF7HANCNFSM4IJRYKAQ .

imirzadeh commented 5 years ago

Hello,

No. I trained using a teacher assistant(in two steps) But, unfortunately, I'm extremely busy until September 5th and after that, I can share the details with you. I'm really sorry but I have too many works to do in the coming weeks.

Thanks, Iman

TeerathChandani commented 5 years ago

Thanks, I will wait till 5th September