Problem getting output of neural network

[MATTYGILO]

Hi I have a custom model with output shape of [1, 400], that has been quantised with tf.int8 for inputs and outputs. However when I run this code to find the limit of the output it seems unlimited. I am running on esp32.

outputTensor = interpreter.getOutputTensor(0)

num = 0
while num < 10000000:
    print("Testing ", num)
    output = outputTensor.quantizeInt8ToFloat(outputTensor.getValue(num))
    print("Output ", output)
    num += 1

It seems to just keep outputting values even though I feel like should stop after 400

All help is much appreciated thanks

[MATTYGILO]

Ok for anyone confused about the functions class etc these are the parameters.

interpreter: ['class', 'getInputTensor', 'getOutputTensor', 'invoke'] inputTensor: ['class', 'getType', 'getValue', 'quantizeFloatToInt8', 'quantizeInt8ToFloat', 'setValue']

[MATTYGILO]

inputTensor.getType(index) inputTensor.getValue(index) inputTensor.quantizeFloatToInt8(float_value) inputTensor.quantizeInt8ToFloat(int_value) inputTensor.setValue(index, value)

[MATTYGILO]

microlite.interpreter(micro_speech_model_bytes, arena_size, input_callback, output_callback)

[MATTYGILO]

interpreter.getInputTensor(index) interpreter.getOutputTensor(index) interpreter.invoke()

[MATTYGILO]

microlite has these classes/functions

['class', 'name', 'version', 'audio_frontend', 'interpreter', 'tensor']

[MATTYGILO]

inputTensor and outputTensor are classes of microlite.tensor ['class', 'name', 'bases', 'dict', 'getType', 'getValue', 'quantizeFloatToInt8', 'quantizeInt8ToFloat', 'setValue']

[MATTYGILO]

Audio_frontend has these classes/functions ['class', 'name', 'bases', 'dict', 'configure', 'execute']

[MATTYGILO]

Would be great if this could be converted into documentation

[MATTYGILO]

Tensor Arena is the maximum size at any point in time needed to store all the tensors at that moment.

[mocleiri]

@MATTYGILO I can only go by the code you posted. Generally you need to follow these steps:

1. Create the interpreter for the model.
2. Set the value for the input tensor. (a 400 element array)
3. Run inference using (interpreter.invoke())
4. Read the results from the output tensor. ( a 400 element array)

Your code:

outputTensor = interpreter.getOutputTensor(0)

num = 0
while num < 10000000:
    print("Testing ", num)
    output = outputTensor.quantizeInt8ToFloat(outputTensor.getValue(num))
    print("Output ", output)
    num += 1

I don't see you initializing the interpreter, setting the input tensor or invoking the interpreter (i.e. running inference)

The outputTensor should be an array of size 400 so I'm surprised your code works without throwing an array index out of bounds exception. I think this is something I could improve on which is to throw an error if the index requested is out of bounds for the size of the tensor.

I would expect to see something like this:

def noop(interp):
    pass

interpreter = microlite.interpreter(my_model, 2048, noop, noop)

inputTensor = interpreter.getInputTensor(0)
outputTensor = interpreter.getOutputTensor(0)

while True:

    # set the 400 elements of the input tensor
    for i in range(400):
        inputTensor.setValue(i) = something

    interpreter.invoke()

    # read the 400 elements of the output tensor
    for i in range(400):
        print("Output ", i, outputTensor.getValue(i))

The size of your output tensor seems large. Most of the demo models have a small number of predictor outputs. For example microspeech has an input tensor size of [40, 49] essentially [1, 1960] but only 4 outputs predicting yes, no, silence and unknown.

[MATTYGILO]

Thanks @mocleiri I'll paste a bigger clip of my code I initialise when I'm next on my laptop.

Essentially I can run code like this outputTensor.getValue(1000000) And still receive a value E.g -128

I'll look into it more but when I print all values up to 1000000 they seems to output "random" values.

Thanks for the help so far I'll get back to you when I tried some more

[mocleiri]

@MATTYGILO a tensor is an array. Because you expect it to be of shape [1, 400] I expect it to be an array of 400 int8 elements.

You can use the ulab.array to have a different shape but when you set the input tensor or get the output tensor its unrolled into a one dimensional array.

This is the code that runs when you get the ith value of the tensor: https://github.com/mocleiri/tensorflow-micropython-examples/blob/e6d214df0b802b8ad8a371950b9e01c123296d6c/micropython-modules/microlite/tensorflow-microlite.c#L96

Its getting you the ith int8 value.

But its not checking the index retrieved relative to the size of the tensor. I think you are running off the end of the array and because of the huge buffer allocated to the model and tensor arena it doesn't throw an error when it runs off the end of the array.

I will make a change to not let this case occur in the future. I think all values you are reading/writing from 400 to 1000000 are invalid.

Can you print the input and output tensor to confirm their shape?

print (inputTensor)
print(ouputTensor)

That will printout the size and type of each: https://github.com/mocleiri/tensorflow-micropython-examples/blob/e6d214df0b802b8ad8a371950b9e01c123296d6c/micropython-modules/microlite/tensorflow-microlite.c#L60

[MATTYGILO]

@mocleiri Thanks these are my input and output tensors

inputTensor: tensor(type=INT8, dims->size=4)
outputTensor: tensor(type=INT8, dims->size=2)

[MATTYGILO]

@mocleiri I think you may be onto something as my arena size is very big. If you think past 400 they are incorrect I'll only read the first 400. Thanks a lot for the suggestions and help it is much appreciated.