analogdevicesinc / ai8x-synthesis

Quantization and Synthesis (Device Specific Code Generation) for ADI's MAX78000 and MAX78002 Edge AI Devices
Apache License 2.0
55 stars 47 forks source link

Problem synthesizing the tinierssd svhn model #250

Closed maderix closed 1 year ago

maderix commented 1 year ago

I trained the newly added tinier-ssd model with the included training script(scripts/train_svhn_tinierssd.sh) During synthesis, I'm able to quantize the weights by this command: _python quantize.py ../ai8x-training-latest/logs/2022.07.11-214500_SVHN_tinierssd/qat_best.pth.tar ../ai8x-training-latest/logs/2022.07.11-214500_SVHN_tinierssd/qat_bestq.pth.tar --device MAX78000 -v -c networks/svhn-tinierssd.yaml

But ai8xize script is failing due to below error: aix8ize command:

_python ai8xize.py --test-dir generated_svhn_tinierssd --prefix tinierssd_svhn --checkpoint-file ../ai8x-training-latest/logs/2022.07.11-214500_SVHN_tinierssd/qat_bestq.pth.tar --config-file networks/svhn-tinierssd.yaml --device MAX78000 --compact-data --timer 0 --display-checkpoint --verbose --overwrite --mexpress

log:

Reading ../ai8x-training-latest/logs/2022.07.11-214500_SVHN_tinierssd/qat_best_q.pth.tar to configure network weights...

e2.op.bias
   32    64  (2048, 3, 3)        8    -2 -128  127   18432 base.fire3.op.weight                      (64,)          8  -20  127   64 base.fir
e3.op.bias
   64    64  (4096, 3, 3)        8    -2 -128  127   36864 base.fire4.op.weight                      (64,)          8  -81  104   64 base.fir
e4.op.bias
   64    64  (4096, 3, 3)        8    -2 -128  127   36864 base.fire5.op.weight                      (64,)          8 -116  115   64 base.fir
e5.op.bias
   64    64  (4096, 3, 3)        8    -2 -128  127   36864 base.fire6.op.weight                      (64,)          8 -128  108   64 base.fire6.op.bias
   64   128  (8192, 3, 3)        8    -2 -128  127   73728 base.fire7.op.weight                      (128,)         8 -128  127  128 base.fire7.op.bias
  128    32  (4096, 3, 3)        8    -2 -128  127   36864 base.fire8.op.weight                      (32,)          8 -110  127   32 base.fire8.op.bias
   32    32  (1024, 3, 3)        8    -1 -128  127    9216 base.fire9.op.weight                      (32,)          8  -38  113   32 base.fire9.op.bias
   32    32  (1024, 3, 3)        8    -1 -109  127    9216 base.fire10.op.weight                     (32,)          8   -2   61   32 base.fire10.op.bias
   32    16  (512, 3, 3)         8    -2 -128  127    4608 aux_convs.conv12_1.op.weight              (16,)          8  -41   40   16 aux_convs.conv12_1.op.bias
   16    16  (256, 3, 3)         8    -1 -123  127    2304 aux_convs.conv12_2.op.weight              (16,)          8   -3   88   16 aux_convs.conv12_2.op.bias
   32    16  (512, 3, 3)         8    -4  -64  127    4608 pred_convs.loc_fire8.op.weight            (16,)          8  -44  127   16 pred_convs.loc_fire8.op.bias
   32    16  (512, 3, 3)         8    -1 -110   88    4608 pred_convs.loc_fire9.op.weight            (16,)          8  -38   54   16 pred_convs.loc_fire9.op.bias
   32    16  (512, 3, 3)         8    -1  -92  107    4608 pred_convs.loc_fire10.op.weight           (16,)          8  -55   52   16 pred_convs.loc_fire10.op.bias
   16    16  (256, 3, 3)         8    -1 -108  103    2304 pred_convs.loc_conv12_2.op.weight         (16,)          8  -35   34   16 pred_convs.loc_conv12_2.op.bias
   32    44  (1408, 3, 3)        8    -2  -87   97   12672 pred_convs.cl_fire8.op.weight             (44,)          8 -124  127   44 pred_convs.cl_fire8.op.bias
   32    44  (1408, 3, 3)        8    -1 -126  127   12672 pred_convs.cl_fire9.op.weight             (44,)          8  -93  125   44 pred_convs.cl_fire9.op.bias
   32    44  (1408, 3, 3)        8    -1 -124  127   12672 pred_convs.cl_fire10.op.weight            (44,)          8  -75   88   44 pred_convs.cl_fire10.op.bias
   16    44  (704, 3, 3)         8    -1 -128  127    6336 pred_convs.cl_conv12_2.op.weight          (44,)          8  -59   50   44 pred_convs.cl_conv12_2.op.bias
TOTAL: 20 parameter layers, 336,336 parameters, 336,336 bytes
TOTAL: 20 parameter layers, 336,336 parameters, 336,336 bytes
Configuring data set: svhn_74.
tinierssd_svhn...
NOTICE: --overwrite specified, writing to generated_svhn_tinierssd/tinierssd_svhn even though it exists.
Arranging weights... ________________________________________ 100%
Storing weights...   ________________________________________ 100%
Creating network...  ________________________________________  10%**ERROR: Processor 0: Layer 2 output for CHW=0,27,25 is overwriting input at offset 0x00402000 that was created by layer 1, CHW=0,0,0.**

I see the weights.h file is mostly empty. Is there something I'm missing or the config yaml for tinierssd need to be corrected? Creating network... ____ 10%

rotx-maxim commented 1 year ago
  1. Make sure you are on the latest code and then
  2. Take a look at gen-demos-max78000.sh. The line for svhn_tinierssd has an additional command line parameter, --overlap-data:

python ai8xize.py --test-dir $TARGET --prefix svhn_tinierssd --checkpoint-file trained/ai85-svhn-tinierssd-qat8-q.pth.tar --config-file networks/svhn-tinierssd.yaml --overlap-data $COMMON_ARGS "$@"

maderix commented 1 year ago

Yeah that seems to help. Thanks.