search

hustzxd / LSQuantization

The PyTorch implementation of Learned Step size Quantization (LSQ) in ICLR2020 (unofficial)
123 stars 21 forks source link

Hyper parameter for cifar10-vggsmall #2

Open BaptisteNguyen opened 3 years ago

BaptisteNguyen commented 3 years ago

Hello, What are the hyper parameter for the training of vggsmall on cifar 10?

haibao-yu commented 3 years ago

Hello, What are the hyper parameter for the training of vggsmall on cifar 10?

Hi, how is the problem going? I also implement lsq method with resnet20 on cifar, but there is more than 3% Top-1 Accuracy drop.

haibao-yu commented 3 years ago

Hello, What are the hyper parameter for the training of vggsmall on cifar 10?

Hi, how is the problem going? I also implement lsq method with resnet20 on cifar, but there is more than 3% Top-1 Accuracy drop.

For ResNet20 on Cifar10, I have gotten Top-1 Accuracy 90.2% with 2bit for weight and 2bit for activation, compared to 91.8% with full precision. I think the results are quite good, which will be better with further hyperparameters tuning. Following are the experiment details:

  1. to train the full precision as a pre-trained model: I set the initial learning rate as 0.1 with Cosine schedule and 160 epochs, weight decay as 1e-4, batch size as 128
  2. to initialize the step size: I set 1.0 for activation as the README
  3. to train the quantized model: I set the initial learning rate as 0.2 with Cosine schedule and 90 epochs, weight decay as 1e-4, batch size as 512
BaptisteNguyen commented 3 years ago

Hello, Thank you for your answer. I will test this.

        Baptiste Nguyen

De : walk2out [notifications@github.com] Envoyé : samedi 9 janvier 2021 12:58 À : hustzxd/LSQuantization Cc : BaptisteNguyen; Author Objet : Re: [hustzxd/LSQuantization] Hyper parameter for cifar10-vggsmall (#2)

Hello, What are the hyper parameter for the training of vggsmall on cifar 10?

Hi, how is the problem going? I also implement lsq method with resnet20 on cifar, but there is more than 3% Top-1 Accuracy drop.

For ResNet20 on Cifar10, I have gotten Top-1 Accuracy 90.2% with 2bit for weight and 2bit for activation, compared to 91.8% with full precision. I think the results are quite good, which will be better with further hyperparameters tuning. Following are the experiment details:

  1. to train the full precision as a pre-trained model: I set the initial learning rate as 0.1 with Cosine schedule and 160 epochs, weight decay as 1e-4, batch size as 128
  2. to initialize the step size: I set 1.0 for activation as the README
  3. to train the quantized model: I set the initial learning rate as 0.2 with Cosine schedule and 90 epochs, weight decay as 1e-4, batch size as 512

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/hustzxd/LSQuantization/issues/2#issuecomment-757138917, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQEMV62KVRAL3FRVNVZMUATSZBAHNANCNFSM4TW3UMQQ.