search

leaderj1001 / Mixed-Depthwise-Convolutional-Kernels

Implementing MixNet: Mixed Depthwise Convolutional Kernels using Pytorch
MIT License
61 stars 8 forks source link

Training MixNet models for Cifar 10 #4

Open meteozay opened 4 years ago

meteozay commented 4 years ago

Hello,

Have you trained MixNet models for Cifar-10 from scratch (not transfer learning from ImageNet models)?

With default configuration, I obtain about 70% accuracy for MixNet small, and the accuracy decreases as the model size increases. Do I need to change some particular parameters in the configuration while training Cifar-10?

Thank you.

leaderj1001 commented 4 years ago

Hi, I'm Myeongjun Kim.

How many epochs are you training? I think It was not completely training.

Thank you.

Best regards,

Myeongjun Kim

On Thu, Nov 7, 2019 at 8:15 AM Mete Ozay notifications@github.com wrote:

Hello,

Have you trained MixNet models for Cifar-10 from scratch (not transfer learning from ImageNet models)?

With default configuration, I obtain about 70% accuracy for MixNet small, and the accuracy decreases as the model size increases. Do I need to change some particular parameters in the configuration while training Cifar-10?

Thank you.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/leaderj1001/Mixed-Depthwise-Convolutional-Kernels/issues/4?email_source=notifications&email_token=AFIOHZR2A3QZBQ7DRDNBIVTQSNFZZA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HXONZAA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIOHZWTLQTSZAPXERN7GR3QSNFZZANCNFSM4JJ7C2FA .

meteozay commented 4 years ago

Hi Myeongjun,

I train usually for 1000 epochs, however, accuracy does not change much after 300 epochs (converges to some points at about 300 epochs).

What is the best accuracy you achieved for Cifar-10 or 100?

Thank you,

Best,

Mete.

On Thu, Nov 7, 2019 at 10:58 AM Myeongjun Kim notifications@github.com wrote:

Hi, I'm Myeongjun Kim.

How many epochs are you training? I think It was not completely training.

Thank you.

Best regards,

Myeongjun Kim

On Thu, Nov 7, 2019 at 8:15 AM Mete Ozay notifications@github.com wrote:

Hello,

Have you trained MixNet models for Cifar-10 from scratch (not transfer learning from ImageNet models)?

With default configuration, I obtain about 70% accuracy for MixNet small, and the accuracy decreases as the model size increases. Do I need to change some particular parameters in the configuration while training Cifar-10?

Thank you.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub < https://github.com/leaderj1001/Mixed-Depthwise-Convolutional-Kernels/issues/4?email_source=notifications&email_token=AFIOHZR2A3QZBQ7DRDNBIVTQSNFZZA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HXONZAA , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AFIOHZWTLQTSZAPXERN7GR3QSNFZZANCNFSM4JJ7C2FA

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/leaderj1001/Mixed-Depthwise-Convolutional-Kernels/issues/4?email_source=notifications&email_token=ABCQZY4RT4Q23ME5Y4XVKNTQSNY4NA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIUDVQ#issuecomment-550584790, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCQZY5EKODEOUG3OBG33OTQSNY4NANCNFSM4JJ7C2FA .

leaderj1001 commented 4 years ago

What is written in the readme is the accuracy report. Or maybe the problem is caused by a different pytorch version. I used the 1.1.0 version.

Thank you.

On Thu, Nov 7, 2019 at 11:29 AM Mete Ozay notifications@github.com wrote:

Hi Myeongjun,

I train usually for 1000 epochs, however, accuracy does not change much after 300 epochs (converges to some points at about 300 epochs).

What is the best accuracy you achieved for Cifar-10 or 100?

Thank you,

Best,

Mete.

On Thu, Nov 7, 2019 at 10:58 AM Myeongjun Kim notifications@github.com wrote:

Hi, I'm Myeongjun Kim.

How many epochs are you training? I think It was not completely training.

Thank you.

Best regards,

Myeongjun Kim

On Thu, Nov 7, 2019 at 8:15 AM Mete Ozay notifications@github.com wrote:

Hello,

Have you trained MixNet models for Cifar-10 from scratch (not transfer learning from ImageNet models)?

With default configuration, I obtain about 70% accuracy for MixNet small, and the accuracy decreases as the model size increases. Do I need to change some particular parameters in the configuration while training Cifar-10?

Thank you.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <

https://github.com/leaderj1001/Mixed-Depthwise-Convolutional-Kernels/issues/4?email_source=notifications&email_token=AFIOHZR2A3QZBQ7DRDNBIVTQSNFZZA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HXONZAA

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AFIOHZWTLQTSZAPXERN7GR3QSNFZZANCNFSM4JJ7C2FA

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/leaderj1001/Mixed-Depthwise-Convolutional-Kernels/issues/4?email_source=notifications&email_token=ABCQZY4RT4Q23ME5Y4XVKNTQSNY4NA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIUDVQ#issuecomment-550584790 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ABCQZY5EKODEOUG3OBG33OTQSNY4NANCNFSM4JJ7C2FA

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/leaderj1001/Mixed-Depthwise-Convolutional-Kernels/issues/4?email_source=notifications&email_token=AFIOHZXMLRRDJ55AY55RC6DQSN4P7A5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIV4GA#issuecomment-550592024, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIOHZQCXQVBPZQZ7RK77OTQSN4P7ANCNFSM4JJ7C2FA .

meteozay commented 4 years ago

Thank you.

Update:

I implemented various other code of MixNets provided in different projects to train MixNet models using Cifar-10. The best achieved accuracy is about 80%. Imo, the problem is algorithmic , i.e. training MixNet from scratch does not perform well, for Cifar 10. Alternatively, there are some "implementation tricks" missed in these code.

Please let me know the details, if you can succeed to achieve >= 92% accuracy by training MixNets using Cifar-10 from scratch.

Best,

Mete.

On Thu, Nov 7, 2019 at 1:14 PM Myeongjun Kim notifications@github.com wrote:

What is written in the readme is the accuracy report. Or maybe the problem is caused by a different pytorch version. I used the 1.1.0 version.

Thank you.

On Thu, Nov 7, 2019 at 11:29 AM Mete Ozay notifications@github.com wrote:

Hi Myeongjun,

I train usually for 1000 epochs, however, accuracy does not change much after 300 epochs (converges to some points at about 300 epochs).

What is the best accuracy you achieved for Cifar-10 or 100?

Thank you,

Best,

Mete.

On Thu, Nov 7, 2019 at 10:58 AM Myeongjun Kim notifications@github.com wrote:

Hi, I'm Myeongjun Kim.

How many epochs are you training? I think It was not completely training.

Thank you.

Best regards,

Myeongjun Kim

On Thu, Nov 7, 2019 at 8:15 AM Mete Ozay notifications@github.com wrote:

Hello,

Have you trained MixNet models for Cifar-10 from scratch (not transfer learning from ImageNet models)?

With default configuration, I obtain about 70% accuracy for MixNet small, and the accuracy decreases as the model size increases. Do I need to change some particular parameters in the configuration while training Cifar-10?

Thank you.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <

https://github.com/leaderj1001/Mixed-Depthwise-Convolutional-Kernels/issues/4?email_source=notifications&email_token=AFIOHZR2A3QZBQ7DRDNBIVTQSNFZZA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HXONZAA

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AFIOHZWTLQTSZAPXERN7GR3QSNFZZANCNFSM4JJ7C2FA

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <

https://github.com/leaderj1001/Mixed-Depthwise-Convolutional-Kernels/issues/4?email_source=notifications&email_token=ABCQZY4RT4Q23ME5Y4XVKNTQSNY4NA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIUDVQ#issuecomment-550584790

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ABCQZY5EKODEOUG3OBG33OTQSNY4NANCNFSM4JJ7C2FA

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/leaderj1001/Mixed-Depthwise-Convolutional-Kernels/issues/4?email_source=notifications&email_token=AFIOHZXMLRRDJ55AY55RC6DQSN4P7A5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIV4GA#issuecomment-550592024 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AFIOHZQCXQVBPZQZ7RK77OTQSN4P7ANCNFSM4JJ7C2FA

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/leaderj1001/Mixed-Depthwise-Convolutional-Kernels/issues/4?email_source=notifications&email_token=ABCQZY3WXFTHG2QUTWMQQPTQSOIZPA5CNFSM4JJ7C2FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDJP3ZA#issuecomment-550698468, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCQZY5KBA4QLMT4A423RFLQSOIZPANCNFSM4JJ7C2FA .

sporthand commented 4 years ago

Hi, I also have the same problem which my Mixnet was originally installed from pip install Keras_Mixnet. I wonder if your code is similar to Keras_Mixnet.

So how many epoch did you set to get the over 90% of validation accuracy? I initially set the number of epoch as 10,000 and the number of patient as 50. However, it stopped between from 110th to 120 epoch. So, the result is 73% of validation accuracy. But, I am planning to change the number of patient from 50 to 100 in order to get the same validation accuracy. I am looking forward to have your instruction. Thank you!

leaderj1001 commented 4 years ago

The code currently uploaded to github is model architecture in the ImageNet dataset. ImageNet and CIFAR have different input image sizes, so you need to train by modifying the stem layer and model architecture. Thank you.