search

CEA-LIST / N2D2

N2D2 is an open source CAD framework for Deep Neural Network simulation and full DNN-based applications building.
Other
146 stars 35 forks source link

Multiply layer is not handled in ONNX import #86

Closed FabriceAuz closed 3 years ago

FabriceAuz commented 3 years ago

I have a network with a layer that multiplies the output from the previous layer by a constant. This network is saved in an onnx file using pyTorch and read in n2d2. This mult layer is present at different places in the network. When importing and running the network, I get an error message, stating:

Error for testing: AffineTransformation: value matrix for operator Multiplies has size 0x0, whereas data size is 224x224
Continue...

Find below the complete output from n2d2:

$ ./n2d2 ISIC_quant_onnx_N2D2.ini -seed 1 -test -w /dev/null
Option -seed: N2D2 random seed (0 = time based) [1]
Option -test: perform testing
Option -w: start with weights imported from a specified location (even when loading a previously saved state) [/dev/null]
Loading network configuration file ISIC_quant_onnx_N2D2.ini
Warning: overriding property Ignore (previous value was "softmax16") in section [onnx] in INI file ISIC_quant_onnx_N2D2.ini
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/training"
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/training/0"
Found 3322 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/training/1"
Found 13497 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/training/2"
Found 15920 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/training/3"
Found 16547 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/training/4"
Found 18571 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/training/5"
Found 18720 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/training/6"
Found 18883 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/training/7"
Found 19331 stimuli
Found 19331 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/validation"
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/validation/0"
Found 19531 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/validation/1"
Found 19981 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/validation/2"
Found 20131 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/validation/3"
Found 20171 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/validation/4"
Found 20271 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/validation/5"
Found 20286 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/validation/6"
Found 20301 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/validation/7"
Found 20331 stimuli
Found 20331 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/test"
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/test/0"
Found 21331 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/test/1"
Found 23581 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/test/2"
Found 24331 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/test/3"
Found 24531 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/test/4"
Found 25031 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/test/5"
Found 25106 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/test/6"
Found 25181 stimuli
Loading directory database "/local2/is148265/DATABASE/ISIC_N2D2/isic_classification/images_classes/test/7"
Found 25331 stimuli
Found 25331 stimuli
Importing ONNX model:
  ir_version = 6
  producer_name = pytorch
  producer_version = 1.7
  domain =
  model_version = 0
  doc_string =
Layer: 33 [Conv]
CUDNN initialized
  # Shared synapses: 1792
  # Virtual synapses: 86188864
  # Inputs dims: 224 224 3
  # Outputs dims: 224 224 64
Layer: 34 [Constant]
Layer: 35 [Mul]
Layer: 36 [Relu]
  36 -> 35
Layer: 37 [Conv]
Notice: Could not open configuration file: onnx:Conv_def.cfg
  # Shared synapses: 36928
  # Virtual synapses: 1838694464
  # Inputs dims: 224 224 64
  # Outputs dims: 224 224 64
Layer: 38 [Constant]
Layer: 39 [Mul]
Layer: 40 [Relu]
  40 -> 39
Layer: 41 [MaxPool]
  # Inputs dims: 224 224 64
  # Outputs dims: 112 112 64
Layer: 42 [Conv]
Notice: Could not open configuration file: onnx:Conv_def.cfg
  # Shared synapses: 73856
  # Virtual synapses: 913866880
  # Inputs dims: 112 112 64
  # Outputs dims: 112 112 128
Layer: 43 [Constant]
Layer: 44 [Mul]
Layer: 45 [Relu]
  45 -> 44
Layer: 46 [Conv]
Notice: Could not open configuration file: onnx:Conv_def.cfg
  # Shared synapses: 147584
  # Virtual synapses: 1827733632
  # Inputs dims: 112 112 128
  # Outputs dims: 112 112 128
Layer: 47 [Constant]
Layer: 48 [Mul]
Layer: 49 [Relu]
  49 -> 48
Layer: 50 [MaxPool]
Notice: Could not open configuration file: onnx:Pool_def.cfg
  # Inputs dims: 112 112 128
  # Outputs dims: 56 56 128
Layer: 51 [Conv]
Notice: Could not open configuration file: onnx:Conv_def.cfg
  # Shared synapses: 295168
  # Virtual synapses: 902955264
  # Inputs dims: 56 56 128
  # Outputs dims: 56 56 256
Layer: 52 [Constant]
Layer: 53 [Mul]
Layer: 54 [Relu]
  54 -> 53
Layer: 55 [Conv]
Notice: Could not open configuration file: onnx:Conv_def.cfg
  # Shared synapses: 590080
  # Virtual synapses: 1805910272
  # Inputs dims: 56 56 256
  # Outputs dims: 56 56 256
Layer: 56 [Constant]
Layer: 57 [Mul]
Layer: 58 [Relu]
  58 -> 57
Layer: 59 [Conv]
Notice: Could not open configuration file: onnx:Conv_def.cfg
  # Shared synapses: 590080
  # Virtual synapses: 1805910272
  # Inputs dims: 56 56 256
  # Outputs dims: 56 56 256
Layer: 60 [Constant]
Layer: 61 [Mul]
Layer: 62 [Relu]
  62 -> 61
Layer: 63 [MaxPool]
Notice: Could not open configuration file: onnx:Pool_def.cfg
  # Inputs dims: 56 56 256
  # Outputs dims: 28 28 256
Layer: 64 [Conv]
Notice: Could not open configuration file: onnx:Conv_def.cfg
  # Shared synapses: 1180160
  # Virtual synapses: 881328640
  # Inputs dims: 28 28 256
  # Outputs dims: 28 28 512
Layer: 65 [Constant]
Layer: 66 [Mul]
Layer: 67 [Relu]
  67 -> 66
Layer: 68 [Conv]
Notice: Could not open configuration file: onnx:Conv_def.cfg
  # Shared synapses: 2359808
  # Virtual synapses: 1762656768
  # Inputs dims: 28 28 512
  # Outputs dims: 28 28 512
Layer: 69 [Constant]
Layer: 70 [Mul]
Layer: 71 [Relu]
  71 -> 70
Layer: 72 [Conv]
Notice: Could not open configuration file: onnx:Conv_def.cfg
  # Shared synapses: 2359808
  # Virtual synapses: 1762656768
  # Inputs dims: 28 28 512
  # Outputs dims: 28 28 512
Layer: 73 [Constant]
Layer: 74 [Mul]
Layer: 75 [Relu]
  75 -> 74
Layer: 76 [MaxPool]
Notice: Could not open configuration file: onnx:Pool_def.cfg
  # Inputs dims: 28 28 512
  # Outputs dims: 14 14 512
Layer: 77 [Conv]
Notice: Could not open configuration file: onnx:Conv_def.cfg
  # Shared synapses: 2359808
  # Virtual synapses: 419430912
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: 78 [Constant]
Layer: 79 [Mul]
Layer: 80 [Relu]
  80 -> 79
Layer: 81 [Conv]
Notice: Could not open configuration file: onnx:Conv_def.cfg
  # Shared synapses: 2359808
  # Virtual synapses: 419430912
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: 82 [Constant]
Layer: 83 [Mul]
Layer: 84 [Relu]
  84 -> 83
Layer: 85 [Conv]
Notice: Could not open configuration file: onnx:Conv_def.cfg
  # Shared synapses: 2359808
  # Virtual synapses: 419430912
  # Inputs dims: 14 14 512
  # Outputs dims: 14 14 512
Layer: 86 [Constant]
Layer: 87 [Mul]
Layer: 88 [Relu]
  88 -> 87
Layer: 89 [MaxPool]
Notice: Could not open configuration file: onnx:Pool_def.cfg
  # Inputs dims: 14 14 512
  # Outputs dims: 7 7 512
Layer: 90 [AveragePool]
Notice: Could not open configuration file: onnx:Pool_def.cfg
  # Inputs dims: 7 7 512
  # Outputs dims: 7 7 512
Layer: 91 [Flatten]
  Ignore Flatten operation
  91 -> 90
Layer: 92 [Gemm]
  # Synapses: 102764544
  # Inputs dims: 7 7 512
  # Outputs dims: 1 1 4096
Layer: 93 [Constant]
Layer: 94 [Mul]
Layer: 95 [Relu]
  95 -> 94
Layer: 96 [Gemm]
Notice: Could not open configuration file: onnx:Fc_def.cfg
  # Synapses: 16781312
  # Inputs dims: 1 1 4096
  # Outputs dims: 1 1 4096
Layer: 97 [Constant]
Layer: 98 [Mul]
Layer: 99 [Relu]
  99 -> 98
Layer: 100 [Gemm]
Notice: Could not open configuration file: onnx:Fc_def.cfg
  # Synapses: 32776
  # Inputs dims: 1 1 4096
  # Outputs dims: 1 1 8
Layer: 101 [Constant]
Layer: softmax16 [Mul]
Target: 100 (target value: 1 / default value: 0 / top-n value: 1)
Total number of neurons: 13555720
Total number of nodes: 15111176
Total number of synapses: 134293320
Total number of virtual synapses: 14965773192
Total number of connections: 14971919752
Notice: Unused section 100.Target in INI file
Notice: Unused section 33 in INI file
Notice: Unused section onnx:Conv_def in INI file
Notice: Unused section 37 in INI file
Notice: Unused section 41 in INI file
Notice: Unused section onnx:Pool_def in INI file
Notice: Unused section 42 in INI file
Notice: Unused section 46 in INI file
Notice: Unused section 50 in INI file
Notice: Unused section 51 in INI file
Notice: Unused section 55 in INI file
Notice: Unused section 59 in INI file
Notice: Unused section 63 in INI file
Notice: Unused section 64 in INI file
Notice: Unused section 68 in INI file
Notice: Unused section 72 in INI file
Notice: Unused section 76 in INI file
Notice: Unused section 77 in INI file
Notice: Unused section 81 in INI file
Notice: Unused section 85 in INI file
Notice: Unused section 89 in INI file
Notice: Unused section 90 in INI file
Notice: Unused section 92 in INI file
Notice: Unused section onnx:Fc_def in INI file
Notice: Unused section 96 in INI file
Notice: Unused section 100 in INI file
Learning database size: 19331 images
Validation database size: 1000 images
Testing database size: 5000 images
Notice: stimuli depth is 8U (according to database first stimulus)
[LOG] Stimuli transformations flow (transformations.png)
[LOG] Network graph (ISIC_quant_onnx_N2D2.ini.png)
[LOG] Network SVG graph (ISIC_quant_onnx_N2D2.ini.svg)
[LOG] Network stats (stats/*)
[LOG] Solvers scheduling (schedule/*)
[LOG] Layer's receptive fields (receptive_fields.log)
[LOG] Labels mapping (*.Target/labels_mapping.log)
[LOG] Labels legend (*.Target/labels_legend.png)
[LOG] Learn frame samples (frames/frame*)
[LOG] Test frame samples (frames/test_frame*)
Error for testing: AffineTransformation: value matrix for operator Multiplies has size 0x0, whereas data size is 224x224
Continue...
Time elapsed: 625.542 s
olivierbichler-cea commented 3 years ago

This is now fixed in the latest commit: the ONNX Mul layer is implemented in N2D2 with ScalingCell for constant Mul.