simple convolution outputs not matching with hand calculation and cmsis-nn simp_conv function outputs

[brainchip-india]

Hi @meenchen @RaymondWang0, context: I am trying to run a simple convolution layer (by using convolve_s8_kernel3_stride1_pad1 function) and trying to compare the outputs with hand calculation and also with cmsis-nn simple convolution outputs, also trying to compare the timing results for the layer.

Issue: The outputs of (convolve_s8_kernel3_stride1_pad1 function) are not matching with the hand calculated outputs please refer to the attached code snippet below.

FYI: Hand calculated outputs got matched with cmsis-nn simple convolution function.

My question is am I calling correct simple convolution function or is there any other simple convolution function to test the basic convolution using mcunet kernel.

`

include "arm_math.h"

include "arm_nnfunctions.h"

include "arm_nnsupportfunctions.h"

include "img2col_element.h"

include "tinyengine_function.h"

include

define CONV_WT_M4 {0, -1, 1, -1, 0, -1, 0, -1, -1, 0, 1, 1, -1, 1, 1, 0, 1, -1, 1, -1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, -1, 1, 0, 0, 0, -1, -1, 1, -1, -1, -1, 1, 1, 1, 1, 1, -1, -1, -1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, -1, -1, -1, 0, 0, 0, 0, 0, 0}

const int8_t in_data[36] = { 1, 2, 0, 2, 1, 2, 1, 0, 1, 2, 2, 2, 2, 1, 0, 2, 2, 0, 2, 2, 0, 0, 1, 2, 2, 0, 2, 0, 0, 1, 0, 0, 1, 0, 1, 0 };

define CONV_BIAS_M4 {0, 0}

const int8_t out_data[18] = { -2, -1, 4, -2, 4, -1, -1, 7, 1, -1, -5, -2, 3, 3, -2, -3, 1, 1 };

define CONV_IN_DIM_M4 3

define CONV_IN_CH_M4 4

define CONV_KER_DIM_M4 3

define CONV_PAD_M4 1

define CONV_STRIDE_M4 1

define CONV_OUT_CH_M4 2

define CONV_OUT_DIM_M4 3

define CONV_BIAS_LSHIFT_M4 0

define CONV_OUT_RSHIFT_M4 0

static const q7_t conv2_wt[CONV_IN_CH_M4 CONV_KER_DIM_M4 CONV_KER_DIM_M4 * CONV_OUT_CH_M4] = CONV_WT_M4;

static const q7_t conv2_bias[CONV_OUT_CH_M4] = CONV_BIAS_M4;

q7_t scratch_buffer_2[92160];

q15_t buffer2 = (q15_t ) scratch_buffer_2;

q7_t output_data[CONV_OUT_DIM_M4 CONV_OUT_DIM_M4 CONV_OUT_CH_M4];

static const q7_t conv2_out_mul[CONV_OUT_CH_M4] = { 1, 1};

static const q7_t conv2_out_shift[CONV_OUT_CH_M4] = { 30, 30 };

int success_or_not = !tinyengine_status;

int status = convolve_s8_kernel3_stride1_pad1 ((const q7_t *) in_data, CONV_IN_DIM_M4, CONV_IN_DIM_M4, CONV_IN_CH_M4, conv2_wt,

conv2_bias, conv2_out_shift, conv2_out_mul, 0, 0, -128, 127, (q7_t *) output_data, CONV_OUT_DIM_M4, CONV_OUT_DIM_M4, CONV_OUT_CH_M4, buffer2, 0);

if (success_or_not != status) {

printf ("Function call Failed\r\n");

} else {

printf ("Function call Passed\r\n");

}

`

[brainchip-india]

Expected outputs: channel 0 -2 -1 4 -2 4 -1 -1 7 1 channel 1 -1 -5 -2 3 3 -2 -3 1 1 ############################################# MCU simp conv outputs channel 0 0 1 0 1 0 1 0 1 0 channel 1 1 0 1 0 1 0 1 0 1

[brainchip-india]

Hi @meenchen @RaymondWang0, can you please look into it and help me on this.

[brainchip-india]

Hi @meenchen @RaymondWang0, after adjusting the multiplier to static const q7_t conv2_out_mul[CONV_OUT_CH_M4] = {2, 2}; I see that the outputs are getting almost matching with the hand calculated outputs.

can u please check any other variable I need to modify.

Updated MCU outputs:

MCU OUTPUTS channel 0 -2 -1 0 -2 0 -1 -1 -1 1 channel 1 -1 -1 -2 -1 -1 -2 1 1 1

[meenchen]

Hi @brainchip-india,

Sorry for the late reply and thanks a lot for reporting this issue. This kernel is not covered in our test since we usually use convolve_s8_kernel2x3_inputch3_stride2_pad1.c in our models. We will check this out and see what's the issue.

[brainchip-india]

define CONV_IN_DIM_M4 6

define CONV_IN_CH_M4 4

define CONV_KER_DIM_M4 3

define CONV_PAD_M4 1

define CONV_STRIDE_M4 1

define CONV_OUT_CH_M4 2

define CONV_OUT_DIM_M4 6

define CONV_BIAS_LSHIFT_M4 0

define CONV_OUT_RSHIFT_M4 0

define CONV_OUT_SHIFT_MCU 27

fixed the issue with the above parameters, understood what's going on inside @meenchen thanks for the help feel free to reopen if anyone facing the issue.

[johnk991]

Hi there, I tried running a similar example on STM32F411RE. The driver code is: `

include "arm_math.h"

include "arm_nnfunctions.h"

include "arm_nnsupportfunctions.h"

include "img2col_element.h"

include "tinyengine_function.h"

include

define CONV_WT_M4 {0, -1, 1, -1, 0, -1, 0, -1, -1, 0, 1, 1, -1, 1, 1, 0, 1, -1, 1, -1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, -1, 1, 0, 0, 0, -1, -1, 1, -1, -1, -1, 1, 1, 1, 1, 1, -1, -1, -1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, -1, -1, -1, 0, 0, 0, 0, 0, 0}

const int8_t in_data[36] = { 1, 2, 0, 2, 1, 2, 1, 0, 1, 2, 2, 2, 2, 1, 0, 2, 2, 0, 2, 2, 0, 0, 1, 2, 2, 0, 2, 0, 0, 1, 0, 0, 1, 0, 1, 0 };

define CONV_BIAS_M4 {0, 0}

const int8_t out_data[18] = { -2, -1, 4, -2, 4, -1, -1, 7, 1, -1, -5, -2, 3, 3, -2, -3, 1, 1 };

define CONV_IN_DIM_M4 6

define CONV_IN_CH_M4 4

define CONV_KER_DIM_M4 3

define CONV_PAD_M4 1

define CONV_STRIDE_M4 1

define CONV_OUT_CH_M4 2

define CONV_OUT_DIM_M4 6

define CONV_BIAS_LSHIFT_M4 0

define CONV_OUT_RSHIFT_M4 0

define CONV_OUT_SHIFT_MCU 27

static const q7_t conv2_wt[CONV_IN_CH_M4 CONV_KER_DIM_M4 CONV_KER_DIM_M4 * CONV_OUT_CH_M4] = CONV_WT_M4;

static const q7_t conv2_bias[CONV_OUT_CH_M4] = CONV_BIAS_M4;

q7_t scratch_buffer_2[92160];

q15_t buffer2 = (q15_t ) scratch_buffer_2;

q7_t output_data[CONV_OUT_DIM_M4 CONV_OUT_DIM_M4 CONV_OUT_CH_M4];

static const q7_t conv2_out_mul[CONV_OUT_CH_M4] = {2, 2};

static const q7_t conv2_out_shift[CONV_OUT_CH_M4] = { 30, 30 };

int main(){ //int success_or_not = !(tinyengine_status.STATE_SUCCESS); int status = convolve_s8_kernel3_stride1_pad1((const q7_t ) in_data, CONV_IN_DIM_M4,CONV_IN_DIM_M4, CONV_IN_CH_M4, conv2_wt,conv2_bias, conv2_out_shift, conv2_out_mul,0, 0, -128, 127, (q7_t ) output_data,CONV_OUT_DIM_M4, CONV_OUT_DIM_M4,CONV_OUT_CH_M4, buffer2, 0);

} ` The output_data buffer at the end of the function call either contains incorrect values (0,127,-128 etc) or I get a hardFault exception. Is there any chance you could provide the code that worked for you as well as the board you tested it on? The results you mentioned should be on the output_data buffer, is that correct?

[brainchip-india]

Hi @johnk991 attaching the main.c and params.h header file, I modified original mcunet main.c so please ignore other code. yes the results posted are from output_data_buffer.

main_code_and_params.zip

[johnk991]

Hi @brainchip-india . Thank you so much! I have 2 questions: 1) Have you understood what the LSHIFT, RSHIFT and OUTSHIFT parameters do? If so could you explain how you configured them? 2) Have you also performed inference to validate the accuracy using test data? If so, is there any chance you could share the code?

[brainchip-india]

Hi @johnk991, for the 1st question I have not gone much into the code for LSHIFT and RSHIFT (I didn't saw any similar params in CMSIS-NN functions so I made those params to value 0), and the OUTSHIFT is how many times u shift the outputs (if you don't know the exact value just brute force with the for loop and find the value for which it satisfies the outputs match and back-trace the code, at the beginning I did the same LoL).

For the 2nd question: No I didn't tried any real model application, just I have validated the test data with CMSIS basic and fast functions and also manually hand calculated the conv layer which is matching with mcunet func outputs and it is already there in the attached zip file.