search

CEA-LIST / N2D2

N2D2 is an open source CAD framework for Deep Neural Network simulation and full DNN-based applications building.
Other
146 stars 36 forks source link

Floating point exception when exporting a network in C #87

Closed FabriceAuz closed 3 years ago

FabriceAuz commented 3 years ago

Although the previous issue was closed, there is still a floating point exception when C-exporting a MobileNet network. Here is the execution trace : 

./n2d2 "./MobileNet_v1_batchnorm_export_C.ini" -seed 1 -test -w weights_validation -export C -fuse -nbbits 8 -calib -1 -act-clipping-mode KL-Divergence -db-export 10 -export-parameters FPGA.ini 

Option -seed: N2D2 random seed (0 = time based) [1]
Option -test: perform testing
Option -fuse: fuse BatchNorm with Conv for test and export
Option -export: generate an export and exit [C]
Option -nbbits: number of bits per weight for exports [8]
Option -calib: number of stimuli used for the calibration (0 = no calibration, -1 = use the full test dataset) [-1]
Option -act-clipping-mode: activation clipping mode on export, can be 'None', 'MSE' or 'KL-Divergence' [KL-Divergence]
Option -w: start with weights imported from a specified location (even when loading a previously saved state) [weights_validation]
Option -db-export: max. number of stimuli to export (0 = no dataset export, -1 = unlimited) [10]
Option -export-parameters: parameters for export [DNeuro1.ini]
Loading network configuration file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv1_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv1_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv2_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv2_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv3_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv3_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv4_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv4_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv5_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv5_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv6_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv6_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_1_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_1_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_2_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_2_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_3_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_3_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_4_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_4_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_5_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_5_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv8_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv8_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv9_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv9_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Notice: Unused section database.validation in INI file
Notice: Unused section database.test in INI file
Notice: Unused section sp in INI file
Notice: Unused section sp.StimuliData-raw in INI file
Notice: Unused section sp.Transformation-1 in INI file
Notice: Unused section sp.Transformation-2 in INI file
Notice: Unused section sp.Transformation-3 in INI file
Notice: Unused section sp.Transformation-4 in INI file
Notice: Unused section sp.Transformation-6 in INI file
Notice: Unused section sp.OnTheFlyTransformation-5 in INI file
Notice: Unused section sp.OnTheFlyTransformation-6 in INI file
Notice: Unused section conv_def in INI file
Notice: Unused section conv1 in INI file
Notice: Unused section bn1 in INI file
Notice: Unused section conv1_3x3_dw in INI file
Notice: Unused section bn1_3x3_dw in INI file
Notice: Unused section conv1_1x1 in INI file
Notice: Unused section bn1_1x1 in INI file
Notice: Unused section conv2_3x3_dw in INI file
Notice: Unused section bn2_3x3_dw in INI file
Notice: Unused section conv2_1x1 in INI file
Notice: Unused section bn2_1x1 in INI file
Notice: Unused section conv3_3x3_dw in INI file
Notice: Unused section bn3_3x3_dw in INI file
Notice: Unused section conv3_1x1 in INI file
Notice: Unused section bn3_1x1 in INI file
Notice: Unused section conv4_3x3_dw in INI file
Notice: Unused section bn4_3x3_dw in INI file
Notice: Unused section conv4_1x1 in INI file
Notice: Unused section bn4_1x1 in INI file
Notice: Unused section conv5_3x3_dw in INI file
Notice: Unused section bn5_3x3_dw in INI file
Notice: Unused section conv5_1x1 in INI file
Notice: Unused section bn5_1x1 in INI file
Notice: Unused section conv6_3x3_dw in INI file
Notice: Unused section bn6_3x3_dw in INI file
Notice: Unused section conv6_1x1 in INI file
Notice: Unused section bn6_1x1 in INI file
Notice: Unused section conv7_1_3x3_dw in INI file
Notice: Unused section bn7_1_3x3_dw in INI file
Notice: Unused section conv7_1_1x1 in INI file
Notice: Unused section bn7_1_1x1 in INI file
Notice: Unused section conv7_2_3x3_dw in INI file
Notice: Unused section bn7_2_3x3_dw in INI file
Notice: Unused section conv7_2_1x1 in INI file
Notice: Unused section bn7_2_1x1 in INI file
Notice: Unused section conv7_3_3x3_dw in INI file
Notice: Unused section bn7_3_3x3_dw in INI file
Notice: Unused section conv7_3_1x1 in INI file
Notice: Unused section bn7_3_1x1 in INI file
Notice: Unused section conv7_4_3x3_dw in INI file
Notice: Unused section bn7_4_3x3_dw in INI file
Notice: Unused section conv7_4_1x1 in INI file
Notice: Unused section bn7_4_1x1 in INI file
Notice: Unused section conv7_5_3x3_dw in INI file
Notice: Unused section bn7_5_3x3_dw in INI file
Notice: Unused section conv7_5_1x1 in INI file
Notice: Unused section bn7_5_1x1 in INI file
Notice: Unused section conv8_3x3_dw in INI file
Notice: Unused section bn8_3x3_dw in INI file
Notice: Unused section conv8_1x1 in INI file
Notice: Unused section bn8_1x1 in INI file
Notice: Unused section conv9_3x3_dw in INI file
Notice: Unused section bn9_3x3_dw in INI file
Notice: Unused section conv9_1x1 in INI file
Notice: Unused section bn9_1x1 in INI file
Notice: Unused section pool in INI file
Notice: Unused section fc in INI file
Notice: Unused section softmax in INI file
Notice: Unused section fc.Target in INI file
Notice: Unused section common.config in INI file
Notice: Unused section fc.config in INI file
Notice: Unused section bn.config in INI file
Time elapsed: 0.00630302 s
Error: Couldn't open database directory: /local2/is148265/DATABASE//isic_classification/training
(base) fa125436@is154584:~/MobileNetV1$ echo $N2D2_DATA
/local2/is148265/DATABASE/
(base) fa125436@is154584:~/MobileNetV1$ export N2D2_DATA = /local/DATABASE
-bash: export: « = » : identifiant non valable
-bash: export: « /local/DATABASE » : identifiant non valable
(base) fa125436@is154584:~/MobileNetV1$ export $N2D2_DATA = /local/DATABASE
-bash: export: « /local2/is148265/DATABASE/ » : identifiant non valable
-bash: export: « = » : identifiant non valable
-bash: export: « /local/DATABASE » : identifiant non valable
(base) fa125436@is154584:~/MobileNetV1$ export N2D2_DATA=/local/DATABASE
(base) fa125436@is154584:~/MobileNetV1$ ./n2d2 "./MobileNet_v1_batchnorm_export_DNeuro.ini" -seed 1 -test -w weights_validation -export C -fuse -nbbits 8 -calib -1 -act-clipping-mode KL-Divergence -db-export 10 -export-parameters DNeuro1.ini
Option -seed: N2D2 random seed (0 = time based) [1]
Option -test: perform testing
Option -fuse: fuse BatchNorm with Conv for test and export
Option -export: generate an export and exit [C]
Option -nbbits: number of bits per weight for exports [8]
Option -calib: number of stimuli used for the calibration (0 = no calibration, -1 = use the full test dataset) [-1]
Option -act-clipping-mode: activation clipping mode on export, can be 'None', 'MSE' or 'KL-Divergence' [KL-Divergence]
Option -w: start with weights imported from a specified location (even when loading a previously saved state) [weights_validation]
Option -db-export: max. number of stimuli to export (0 = no dataset export, -1 = unlimited) [10]
Option -export-parameters: parameters for export [DNeuro1.ini]
Loading network configuration file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv1_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv1_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv2_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv2_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv3_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv3_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv4_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv4_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv5_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv5_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv6_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv6_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_1_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_1_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_2_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_2_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_3_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_3_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_4_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_4_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_5_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv7_5_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv8_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv8_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv9_3x3_dw] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Warning: overriding property ActivationFunction (previous value was "Rectifier") in section [conv9_1x1] in INI file ./MobileNet_v1_batchnorm_export_DNeuro.ini
Loading directory database "/local/DATABASE/isic_classification/training"
Loading directory database "/local/DATABASE/isic_classification/training/0"
Found 3322 stimuli
Loading directory database "/local/DATABASE/isic_classification/training/1"
Found 13497 stimuli
Loading directory database "/local/DATABASE/isic_classification/training/2"
Found 15920 stimuli
Loading directory database "/local/DATABASE/isic_classification/training/3"
Found 16547 stimuli
Loading directory database "/local/DATABASE/isic_classification/training/4"
Found 18571 stimuli
Loading directory database "/local/DATABASE/isic_classification/training/5"
Found 18720 stimuli
Loading directory database "/local/DATABASE/isic_classification/training/6"
Found 18883 stimuli
Loading directory database "/local/DATABASE/isic_classification/training/7"
Found 19331 stimuli
Found 19331 stimuli
Loading directory database "/local/DATABASE/isic_classification/validation"
Loading directory database "/local/DATABASE/isic_classification/validation/0"
Found 19531 stimuli
Loading directory database "/local/DATABASE/isic_classification/validation/1"
Found 19981 stimuli
Loading directory database "/local/DATABASE/isic_classification/validation/2"
Found 20131 stimuli
Loading directory database "/local/DATABASE/isic_classification/validation/3"
Found 20171 stimuli
Loading directory database "/local/DATABASE/isic_classification/validation/4"
Found 20271 stimuli
Loading directory database "/local/DATABASE/isic_classification/validation/5"
Found 20286 stimuli
Loading directory database "/local/DATABASE/isic_classification/validation/6"
Found 20301 stimuli
Loading directory database "/local/DATABASE/isic_classification/validation/7"
Found 20331 stimuli
Found 20331 stimuli
Loading directory database "/local/DATABASE/isic_classification/test"
Loading directory database "/local/DATABASE/isic_classification/test/0"
Found 21331 stimuli
Loading directory database "/local/DATABASE/isic_classification/test/1"
Found 23581 stimuli
Loading directory database "/local/DATABASE/isic_classification/test/2"
Found 24331 stimuli
Loading directory database "/local/DATABASE/isic_classification/test/3"
Found 24531 stimuli
Loading directory database "/local/DATABASE/isic_classification/test/4"
Found 25031 stimuli
Loading directory database "/local/DATABASE/isic_classification/test/5"
Found 25106 stimuli
Loading directory database "/local/DATABASE/isic_classification/test/6"
Found 25181 stimuli
Loading directory database "/local/DATABASE/isic_classification/test/7"
Found 25331 stimuli
Found 25331 stimuli
sp.StimuliData-raw processing 19331 stimuli
---
sp.StimuliData-raw data:
Number of stimuli: 19331
Data width range: [576, 1024]
Data height range: [450, 1024]
Data channels range: [3, 3]
Value range: [0, 255]
Value mean: 140.381
Value std. dev.: 60.6431
---
Layer: conv1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv1.cfg
  # Shared synapses: 864
  # Virtual synapses: 10773600
  # Inputs dims: 224 224 3
  # Outputs dims: 112 112 32
Layer: bn1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn1.cfg
  # Inputs dims: 112 112 32
  # Outputs dims: 112 112 32
Layer: conv1_3x3_dw [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv1_3x3_dw.cfg
  # Shared synapses: 288
  # Virtual synapses: 3569792
  # Inputs dims: 112 112 32
  # Outputs dims: 112 112 32
Layer: bn1_3x3_dw [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn1_3x3_dw.cfg
  # Inputs dims: 112 112 32
  # Outputs dims: 112 112 32
Layer: conv1_1x1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv1_1x1.cfg
  # Shared synapses: 2048
  # Virtual synapses: 25690112
  # Inputs dims: 112 112 32
  # Outputs dims: 112 112 64
Layer: bn1_1x1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn1_1x1.cfg
  # Inputs dims: 112 112 64
  # Outputs dims: 112 112 64
Layer: conv2_3x3_dw [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv2_3x3_dw.cfg
  # Shared synapses: 576
  # Virtual synapses: 1784896
  # Inputs dims: 112 112 64
  # Outputs dims: 56 56 64
Layer: bn2_3x3_dw [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn2_3x3_dw.cfg
  # Inputs dims: 56 56 64
  # Outputs dims: 56 56 64
Layer: conv2_1x1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv2_1x1.cfg
  # Shared synapses: 8192
  # Virtual synapses: 25690112
  # Inputs dims: 56 56 64
  # Outputs dims: 56 56 128
Layer: bn2_1x1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn2_1x1.cfg
  # Inputs dims: 56 56 128
  # Outputs dims: 56 56 128
Layer: conv3_3x3_dw [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv3_3x3_dw.cfg
  # Shared synapses: 1152
  # Virtual synapses: 3527168
  # Inputs dims: 56 56 128
  # Outputs dims: 56 56 128
Layer: bn3_3x3_dw [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn3_3x3_dw.cfg
  # Inputs dims: 56 56 128
  # Outputs dims: 56 56 128
Layer: conv3_1x1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv3_1x1.cfg
  # Shared synapses: 16384
  # Virtual synapses: 51380224
  # Inputs dims: 56 56 128
  # Outputs dims: 56 56 128
Layer: bn3_1x1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn3_1x1.cfg
  # Inputs dims: 56 56 128
  # Outputs dims: 56 56 128
Layer: conv4_3x3_dw [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv4_3x3_dw.cfg
  # Shared synapses: 1152
  # Virtual synapses: 881792
  # Inputs dims: 56 56 128
  # Outputs dims: 28 28 128
Layer: bn4_3x3_dw [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn4_3x3_dw.cfg
  # Inputs dims: 28 28 128
  # Outputs dims: 28 28 128
Layer: conv4_1x1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv4_1x1.cfg
  # Shared synapses: 32768
  # Virtual synapses: 25690112
  # Inputs dims: 28 28 128
  # Outputs dims: 28 28 256
Layer: bn4_1x1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn4_1x1.cfg
  # Inputs dims: 28 28 256
  # Outputs dims: 28 28 256
Layer: conv5_3x3_dw [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv5_3x3_dw.cfg
  # Shared synapses: 2304
  # Virtual synapses: 1721344
  # Inputs dims: 28 28 256
  # Outputs dims: 28 28 256
Layer: bn5_3x3_dw [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn5_3x3_dw.cfg
  # Inputs dims: 28 28 256
  # Outputs dims: 28 28 256
Layer: conv5_1x1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv5_1x1.cfg
  # Shared synapses: 65536
  # Virtual synapses: 51380224
  # Inputs dims: 28 28 256
  # Outputs dims: 28 28 256
Layer: bn5_1x1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn5_1x1.cfg
  # Inputs dims: 28 28 256
  # Outputs dims: 28 28 256
Layer: conv6_3x3_dw [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv6_3x3_dw.cfg
  # Shared synapses: 2304
  # Virtual synapses: 430336
  # Inputs dims: 28 28 256
  # Outputs dims: 14 14 256
Layer: bn6_3x3_dw [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn6_3x3_dw.cfg
  # Inputs dims: 14 14 256
  # Outputs dims: 14 14 256
Layer: conv6_1x1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv6_1x1.cfg
  # Shared synapses: 131072
  # Virtual synapses: 25690112
  # Inputs dims: 14 14 256
  # Outputs dims: 14 14 512
Layer: bn6_1x1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn6_1x1.cfg
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: conv7_1_3x3_dw [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv7_1_3x3_dw.cfg
  # Shared synapses: 4608
  # Virtual synapses: 819200
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: bn7_1_3x3_dw [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn7_1_3x3_dw.cfg
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: conv7_1_1x1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv7_1_1x1.cfg
  # Shared synapses: 262144
  # Virtual synapses: 51380224
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: bn7_1_1x1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn7_1_1x1.cfg
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: conv7_2_3x3_dw [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv7_2_3x3_dw.cfg
  # Shared synapses: 4608
  # Virtual synapses: 819200
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: bn7_2_3x3_dw [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn7_2_3x3_dw.cfg
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: conv7_2_1x1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv7_2_1x1.cfg
  # Shared synapses: 262144
  # Virtual synapses: 51380224
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: bn7_2_1x1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn7_2_1x1.cfg
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: conv7_3_3x3_dw [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv7_3_3x3_dw.cfg
  # Shared synapses: 4608
  # Virtual synapses: 819200
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: bn7_3_3x3_dw [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn7_3_3x3_dw.cfg
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: conv7_3_1x1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv7_3_1x1.cfg
  # Shared synapses: 262144
  # Virtual synapses: 51380224
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: bn7_3_1x1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn7_3_1x1.cfg
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: conv7_4_3x3_dw [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv7_4_3x3_dw.cfg
  # Shared synapses: 4608
  # Virtual synapses: 819200
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: bn7_4_3x3_dw [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn7_4_3x3_dw.cfg
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: conv7_4_1x1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv7_4_1x1.cfg
  # Shared synapses: 262144
  # Virtual synapses: 51380224
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: bn7_4_1x1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn7_4_1x1.cfg
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: conv7_5_3x3_dw [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv7_5_3x3_dw.cfg
  # Shared synapses: 4608
  # Virtual synapses: 819200
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: bn7_5_3x3_dw [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn7_5_3x3_dw.cfg
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: conv7_5_1x1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv7_5_1x1.cfg
  # Shared synapses: 262144
  # Virtual synapses: 51380224
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: bn7_5_1x1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn7_5_1x1.cfg
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: conv8_3x3_dw [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv8_3x3_dw.cfg
  # Shared synapses: 4608
  # Virtual synapses: 204800
  # Inputs dims: 14 14 512
  # Outputs dims: 7 7 512
Layer: bn8_3x3_dw [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn8_3x3_dw.cfg
  # Inputs dims: 7 7 512
  # Outputs dims: 7 7 512
Layer: conv8_1x1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv8_1x1.cfg
  # Shared synapses: 524288
  # Virtual synapses: 25690112
  # Inputs dims: 7 7 512
  # Outputs dims: 7 7 1024
Layer: bn8_1x1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn8_1x1.cfg
  # Inputs dims: 7 7 1024
  # Outputs dims: 7 7 1024
Layer: conv9_3x3_dw [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv9_3x3_dw.cfg
  # Shared synapses: 9216
  # Virtual synapses: 369664
  # Inputs dims: 7 7 1024
  # Outputs dims: 7 7 1024
Layer: bn9_3x3_dw [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn9_3x3_dw.cfg
  # Inputs dims: 7 7 1024
  # Outputs dims: 7 7 1024
Layer: conv9_1x1 [Conv(Frame_CUDA)]
Notice: Could not open configuration file: conv9_1x1.cfg
  # Shared synapses: 1048576
  # Virtual synapses: 51380224
  # Inputs dims: 7 7 1024
  # Outputs dims: 7 7 1024
Layer: bn9_1x1 [BatchNorm(Frame_CUDA)]
Notice: Could not open configuration file: bn9_1x1.cfg
  # Inputs dims: 7 7 1024
  # Outputs dims: 7 7 1024
Layer: pool [Pool(Frame_CUDA)]
Notice: Could not open configuration file: pool.cfg
  # Inputs dims: 7 7 1024
  # Outputs dims: 1 1 1024
Layer: fc [Fc(Frame_CUDA)]
Notice: Could not open configuration file: fc.cfg
  # Synapses: 8200
  # Inputs dims: 1 1 1024
  # Outputs dims: 1 1 8
Target: fc (target value: 1 / default value: 0 / top-n value: 5)
Layer: softmax [Softmax(Frame_CUDA)]
Notice: Could not open configuration file: softmax.cfg
  # Inputs dims: 1 1 8
  # Outputs dims: 1 1 8
Total number of neurons: 5042704
Total number of nodes: 10086416
Total number of synapses: 3193288
Total number of virtual synapses: 566859944
Total number of connections: 566910120
Notice: Unused section conv_def in INI file
CUDNN initialized
Learning database size: 19331 images
Validation database size: 1000 images
Testing database size: 5000 images
Notice: stimuli depth is 8U (according to database first stimulus)
Importing weights from directory 'weights_validation'.
Error while reading scale in parameter file: weights_validation/bn1_scales.syntxt
Remove Dropout...
Fuse BatchNorm with Conv...
  fuse BatchNorm "bn1" with Conv "conv1"
    zero-variance conv1[8]
    zero-variance conv1[9]
    zero-variance conv1[10]
    zero-variance conv1[11]
    zero-variance conv1[12]
    zero-variance conv1[13]
    zero-variance conv1[14]
    zero-variance conv1[15]
    zero-variance conv1[16]
    zero-variance conv1[17]
    zero-variance conv1[18]
    zero-variance conv1[19]
    zero-variance conv1[20]
    zero-variance conv1[21]
    zero-variance conv1[22]
    zero-variance conv1[23]
    zero-variance conv1[24]
    zero-variance conv1[25]
    zero-variance conv1[26]
    zero-variance conv1[27]
    zero-variance conv1[28]
    zero-variance conv1[29]
    zero-variance conv1[30]
    zero-variance conv1[31]
  fuse BatchNorm "bn1_1x1" with Conv "conv1_1x1"
    zero-variance conv1_1x1[0]
    zero-variance conv1_1x1[1]
...
   Lots of annoying zero-variance warnings here...
...
    zero-variance conv9_3x3_dw[1019]
    zero-variance conv9_3x3_dw[1020]
    zero-variance conv9_3x3_dw[1021]
    zero-variance conv9_3x3_dw[1022]
    zero-variance conv9_3x3_dw[1023]
    variance < 1e-12 for all outputs! Is the network correctly trained?
export_C_int8/stimuli_stats processing 1000 stimuli...................
Fuse Padding...
  Cross-layer equalization:
    - eq. conv1_3x3_dw and conv1
    - eq. conv1_1x1 and conv1_3x3_dw
    - eq. conv2_3x3_dw and conv1_1x1
    - eq. conv2_1x1 and conv2_3x3_dw
    - eq. conv3_3x3_dw and conv2_1x1
    - eq. conv3_1x1 and conv3_3x3_dw
    - eq. conv4_3x3_dw and conv3_1x1
    - eq. conv4_1x1 and conv4_3x3_dw
    - eq. conv5_3x3_dw and conv4_1x1
    - eq. conv5_1x1 and conv5_3x3_dw
    - eq. conv6_3x3_dw and conv5_1x1
    - eq. conv6_1x1 and conv6_3x3_dw
    - eq. conv7_1_3x3_dw and conv6_1x1
    - eq. conv7_1_1x1 and conv7_1_3x3_dw
    - eq. conv7_2_3x3_dw and conv7_1_1x1
    - eq. conv7_2_1x1 and conv7_2_3x3_dw
    - eq. conv7_3_3x3_dw and conv7_2_1x1
    - eq. conv7_3_1x1 and conv7_3_3x3_dw
    - eq. conv7_4_3x3_dw and conv7_3_1x1
    - eq. conv7_4_1x1 and conv7_4_3x3_dw
    - eq. conv7_5_3x3_dw and conv7_4_1x1
    - eq. conv7_5_1x1 and conv7_5_3x3_dw
    - eq. conv8_3x3_dw and conv7_5_1x1
    - eq. conv8_1x1 and conv8_3x3_dw
    - eq. conv9_3x3_dw and conv8_1x1
    - eq. conv9_1x1 and conv9_3x3_dw
    quant. range delta = 549.388
    - eq. conv1_3x3_dw and conv1
    - eq. conv1_1x1 and conv1_3x3_dw
    - eq. conv2_3x3_dw and conv1_1x1
    - eq. conv2_1x1 and conv2_3x3_dw
    - eq. conv3_3x3_dw and conv2_1x1
    - eq. conv3_1x1 and conv3_3x3_dw
    - eq. conv4_3x3_dw and conv3_1x1
    - eq. conv4_1x1 and conv4_3x3_dw
    - eq. conv5_3x3_dw and conv4_1x1
    - eq. conv5_1x1 and conv5_3x3_dw
    - eq. conv6_3x3_dw and conv5_1x1
    - eq. conv6_1x1 and conv6_3x3_dw
    - eq. conv7_1_3x3_dw and conv6_1x1
    - eq. conv7_1_1x1 and conv7_1_3x3_dw
    - eq. conv7_2_3x3_dw and conv7_1_1x1
    - eq. conv7_2_1x1 and conv7_2_3x3_dw
    - eq. conv7_3_3x3_dw and conv7_2_1x1
    - eq. conv7_3_1x1 and conv7_3_3x3_dw
    - eq. conv7_4_3x3_dw and conv7_3_1x1
    - eq. conv7_4_1x1 and conv7_4_3x3_dw
    - eq. conv7_5_3x3_dw and conv7_4_1x1
    - eq. conv7_5_1x1 and conv7_5_3x3_dw
    - eq. conv8_3x3_dw and conv7_5_1x1
    - eq. conv8_1x1 and conv8_3x3_dw
    - eq. conv9_3x3_dw and conv8_1x1
    - eq. conv9_1x1 and conv9_3x3_dw
    quant. range delta = 151.586
    - eq. conv1_3x3_dw and conv1
    - eq. conv1_1x1 and conv1_3x3_dw
    - eq. conv2_3x3_dw and conv1_1x1
...
    Equalization convergence information...
...
    quant. range delta = 1.00539
    - eq. conv1_3x3_dw and conv1
    - eq. conv1_1x1 and conv1_3x3_dw
    - eq. conv2_3x3_dw and conv1_1x1
    - eq. conv2_1x1 and conv2_3x3_dw
    - eq. conv3_3x3_dw and conv2_1x1
    - eq. conv3_1x1 and conv3_3x3_dw
    - eq. conv4_3x3_dw and conv3_1x1
    - eq. conv4_1x1 and conv4_3x3_dw
    - eq. conv5_3x3_dw and conv4_1x1
    - eq. conv5_1x1 and conv5_3x3_dw
    - eq. conv6_3x3_dw and conv5_1x1
    - eq. conv6_1x1 and conv6_3x3_dw
    - eq. conv7_1_3x3_dw and conv6_1x1
    - eq. conv7_1_1x1 and conv7_1_3x3_dw
    - eq. conv7_2_3x3_dw and conv7_1_1x1
    - eq. conv7_2_1x1 and conv7_2_3x3_dw
    - eq. conv7_3_3x3_dw and conv7_2_1x1
    - eq. conv7_3_1x1 and conv7_3_3x3_dw
    - eq. conv7_4_3x3_dw and conv7_3_1x1
    - eq. conv7_4_1x1 and conv7_4_3x3_dw
    - eq. conv7_5_3x3_dw and conv7_4_1x1
    - eq. conv7_5_1x1 and conv7_5_3x3_dw
    - eq. conv8_3x3_dw and conv7_5_1x1
    - eq. conv8_1x1 and conv8_3x3_dw
    - eq. conv9_3x3_dw and conv8_1x1
    - eq. conv9_1x1 and conv9_3x3_dw
    quant. range delta = 0.984756
export_C_int8/stimuli_stats processing 1000 stimuli
Calculating calibration data range and histogram...
CUBLAS initialized
Floating point exception [floating point invalid operation]
backtrace() returned 7 addresses
./n2d2(_ZN4N2D216exceptionHandlerEiP9siginfo_tPv+0x35)[0x560608682915]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f9fb6b79890]
./n2d2(_ZN4N2D210RangeStatsclEd+0x16)[0x5606085770c6]
./n2d2(+0x8ca41f)[0x5606083ed41f]
/usr/lib/x86_64-linux-gnu/libgomp.so.1(+0x1695e)[0x7f9f91cfd95e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f9fb6b6e6db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f9f917ff88f]
olivierbichler-cea commented 3 years ago

Hi, Some weights are missing during the loading of the network, because the scale factor of MobileNet (alpha) is not the same than the one used during the learning. A warning was issued but it could be easily missed, that's why in the next commit the program will be stopped in such case. Closing.