search

CanyonWind / Single-Path-One-Shot-NAS-MXNet

Single Path One-Shot NAS MXNet implementation with full training and searching pipeline. Support both Block and Channel Selection. Searched models better than the original paper are provided.
151 stars 22 forks source link

Official Review #10

Closed micronet-challenge-submissions closed 5 years ago

micronet-challenge-submissions commented 5 years ago

Hello! Thanks so much for your submission!

When we try to run your model checkpoints we're getting a segmentation fault. We're using the mxnet/python:1.5.0_gpu_cu101_mkl_py3 docker image and running on a V100. We downloaded the dataset from your link. Could you confirm what environment you're evaluating in?

Thanks! Trevor

micronet-challenge-submissions commented 5 years ago

For reference, the error appears to be occurring in the call to ImageRecordIter here:

https://github.com/CanyonWind/Single-Path-One-Shot-NAS-MXNet/blob/461dad7f00b8472d0d5b721098c642d1e690f1d4/MicroNetChallenge/inference.py#L129

CanyonWind commented 5 years ago

Will take a look shortly.

micronet-challenge-submissions commented 5 years ago

Thanks! I switched the call to ImageRecordIter_v1 and it seems to be working now. It still segfaults on GPU though, which is odd. Will update here when inference completes.

micronet-challenge-submissions commented 5 years ago

Just completed - seems to work fine. I see 75.662% top-1 accuracy.

CanyonWind commented 5 years ago

Sorry for the confusion. I just checked. Using GPU as context does cause a segmentation fault. This is because the quantized model is supposed to only run on CPUs. The GPU option for the quantized model needs to be removed @Celia-xy .

Another thing to be noted is that the accuracy would fluctuate because "Accuracy is collected on Intel XEON Cascade Lake CPU. For CPU with Skylake Lake or eariler architecture, the accuracy may not be the same.".

micronet-challenge-submissions commented 5 years ago

Ok sounds good, thanks! A couple of questions about your scoring:

  1. AFAICT there hasn't been a fix for MKL int8, so you'll be submitting your float16 score?
  2. You should count the 'clip' operation in HardSwish as 2 operations, for the min & max.

Otherwise everything looks good!

Trevor

CanyonWind commented 5 years ago

I need to discuss with my partner... Will get back to you later: )

Celia-xy commented 5 years ago

@micronet-challenge-submissions Yes, it seems that float16 is what we have now.

However we are also wondering whether the int8 score can be accepted as our final score since the problem is caused by mxnet mkl-dnn backend. The quantization method they use is the regular int8 quantization with calibration. This is quite common and also implemented in tf/pytorch as a standard procedure.

The current model is a partially quantized one (skipping the problematic operators). We can provide a fully quantized int8 model but the inference result will be wrong because of the mkl-dnn problem. So that the model size would be small and maybe this could help to show that we finished the whole quantization procedure.

This work's contribution is about the full NAS pipeline and the searched efficient structure. We really hope it would not drop out of the leaderboard because of the things we have no control. Looking forward to your decision. Thanks!

micronet-challenge-submissions commented 5 years ago

Thanks for clarifying! It's very unfortunate that this was caused by a bug out of your control, but unfortunately we can't accept your int8 score unless we can validate that the model hits the required quality target.

If you can update your HardSwish counting, everything else checks out on your entry. Thanks!

Trevor

CanyonWind commented 5 years ago

Clip flop has been updated https://github.com/CanyonWind/Single-Path-One-Shot-NAS-MXNet/commit/3ff84e21d6906191db34ad7a1ae12d17b728806b https://github.com/CanyonWind/Single-Path-One-Shot-NAS-MXNet/commit/db5b5eedf48f41976b8fc4e57181d2b6cfde37fd. The new float16 score is 0.640. Guess this is our final score 😂

micronet-challenge-submissions commented 5 years ago

Thanks for the update!

Trevor

On Wed, 30 Oct 2019 at 14:29, Alex notifications@github.com wrote:

Clip flop has been updated 3ff84e2 https://github.com/CanyonWind/Single-Path-One-Shot-NAS-MXNet/commit/3ff84e21d6906191db34ad7a1ae12d17b728806b db5b5ee https://github.com/CanyonWind/Single-Path-One-Shot-NAS-MXNet/commit/db5b5eedf48f41976b8fc4e57181d2b6cfde37fd. The new float16 score is 0.640. Guess this is our final score 😂

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CanyonWind/Single-Path-One-Shot-NAS-MXNet/issues/10?email_source=notifications&email_token=AMILA6225XITWNJAKFEYXRDQRH4FNA5CNFSM4JGQR632YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECV2R2A#issuecomment-548120808, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMILA67VZ2LJ5P2ERP6NVFLQRH4FNANCNFSM4JGQR63Q .

micronet-challenge-submissions commented 5 years ago

Also, what name would you like for us to publish your entry under?

Thanks! Trevor

CanyonWind commented 5 years ago

Please name it Sisyphus. Thanks

micronet-challenge-submissions commented 5 years ago

Sounds good, thanks!

On Fri, 1 Nov 2019 at 16:46, Alex notifications@github.com wrote:

Please name it Sisyphus. Thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CanyonWind/Single-Path-One-Shot-NAS-MXNet/issues/10?email_source=notifications&email_token=AMILA62VPUNZTKOVNAODCCTQRS5TZA5CNFSM4JGQR632YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC4NWWA#issuecomment-548985688, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMILA666HX4CKQRTAC74ZYDQRS5TZANCNFSM4JGQR63Q .