Khadas VIM3 NN enable as TFlite delegate

[RichardPar]

Hi there,

I have managed to get the NN working on the Khadas VIM3 using a TF delegate (libvx_delegate) - I have all the .so's and ko's ready to make it work - just what would be needed to integrate it and perhaps give it a run. (18MB file.. where to upload?)

There is a .ko that needs to be loaded (the version supplied with Khadas VIM3 does not work)

Best Regards, Richard

import numpy as np
import tflite_runtime.interpreter as tflite

# Load TFLite model and allocate tensors.
# (if you are using the complete tensorflow package you can find load_delegate in tf.experimental.load_delegate)
delegate = tflite.load_delegate( library="libvx_delegate.so", options={"logging-severity":"info"})
print(delegate)
# Delegates/Executes all operations supported by Arm NN to/with Arm NN

#interpreter = tflite.Interpreter(model_path="25k_theo_icarus_nano_low-int8.tflite")

interpreter = tflite.Interpreter(model_path="25k_theo_icarus_nano_low-int8.tflite", 
                                 experimental_delegates=[delegate])

interpreter.allocate_tensors()

# Get input and output tensors.
input_details = interpreter.get_input_details()
print("========== INPUT DETAILS ========")
print(input_details)

print()
print("====== OUTPUT DETAILS ==========")
output_details = interpreter.get_output_details()
print(output_details)

# Test model on random input data.
input_shape = input_details[0]['shape']
input_data = np.array(np.random.random_sample(input_shape), dtype=np.uint8)
interpreter.set_tensor(input_details[0]['index'], input_data)

interpreter.invoke()

# Print out result
output_data = interpreter.get_tensor(output_details[0]['index'])
print(output_data)

khadas@Khadas:~/MasterTest$ python3 test.py 
Vx delegate: allowed_cache_mode set to 0.
Vx delegate: device num set to 0.
Vx delegate: allowed_builtin_code set to 0.
Vx delegate: error_during_init set to 0.
Vx delegate: error_during_prepare set to 0.
Vx delegate: error_during_invoke set to 0.
<tflite_runtime.interpreter.Delegate object at 0x7fa444efd0>
========== INPUT DETAILS ========
[{'name': 'serving_default_input_1:0', 'index': 0, 'shape': array([  1, 256, 256,   3], dtype=int32), 'shape_signature': array([  1, 256, 256,   3], dtype=int32), 'dtype': <class 'numpy.uint8'>, 'quantization': (0.003921568859368563, 0), 'quantization_parameters': {'scales': array([0.00392157], dtype=float32), 'zero_points': array([0], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]

====== OUTPUT DETAILS ==========
[{'name': 'StatefulPartitionedCall:0', 'index': 408, 'shape': array([   1, 4032,    6], dtype=int32), 'shape_signature': array([   1, 4032,    6], dtype=int32), 'dtype': <class 'numpy.uint8'>, 'quantization': (0.003994452301412821, 0), 'quantization_parameters': {'scales': array([0.00399445], dtype=float32), 'zero_points': array([0], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
W [HandleLayoutInfer:281]Op 162: default layout inference pass.
[[[  8   8  18  18   0 249]
  [  8   9  18  22   0 249]
  [  7   9  18  22   0 249]
  ...
  [227 229  48  44   0 249]
  [224 221  54  58   0 249]
  [224 224 129 112   0 249]]]

khadas@Khadas:~/MasterTest$ ldd libvx_delegate.so
    linux-vdso.so.1 (0x0000007f97409000)
    libtim-vx.so => /home/khadas/MasterTest/libtim-vx.so (0x0000007f96ad9000)
    librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007f96ac1000)
    libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f96a90000)
    libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f96a7c000)
    libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f969d1000)
    libCLC.so => /home/khadas/MasterTest/libCLC.so (0x0000007f96775000)
    libGAL.so => /home/khadas/MasterTest/libGAL.so (0x0000007f96573000)
    libOpenVX.so.1 => /home/khadas/MasterTest/libOpenVX.so.1 (0x0000007f962ce000)
    libOpenVXU.so => /home/khadas/MasterTest/libOpenVXU.so (0x0000007f962b7000)
    libVSC.so => /home/khadas/MasterTest/libVSC.so (0x0000007f952fe000)
    libArchModelSw.so => /home/khadas/MasterTest/libArchModelSw.so (0x0000007f95294000)
    libNNArchPerf.so => /home/khadas/MasterTest/libNNArchPerf.so (0x0000007f95225000)
    libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f9504d000)
    libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f95029000)
    libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f94eb4000)
    /lib/ld-linux-aarch64.so.1 (0x0000007f973d9000)

[RichardPar]

Anyone able to offer assistance?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[blakeblackshear]

I need to figure out how we want to incorporate support for SBC boards that are used by a small number of users. In the main project, we don't want to sign up for maintaining support for every hardware platform that exists. I have some ideas for community supported boards, but I need more time to think it through.

[RichardPar]

I need to figure out how we want to incorporate support for SBC boards that are used by a small number of users. In the main project, we don't want to sign up for maintaining support for every hardware platform that exists. I have some ideas for community supported boards, but I need more time to think it through.

this will happen on every board that also supports ARM-NNAPI on neural as each Mali GPU needs its appropriate shared libraries

[blakeblackshear]

Exactly. I don't want to hold up progress on Frigate while we fix compatability with dozens of boards that only a handful of users have. If the community wants to maintain those, we can make it a more defined process.

[ACiDGRiM]

Exactly. I don't want to hold up progress on Frigate while we fix compatability with dozens of boards that only a handful of users have. If the community wants to maintain those, we can make it a more defined process.

Maybe if the ffmpeg and associated libraries can be stored in a specific folder in the frigate image, users can just mount a custom volume over that folder. This would take supporting anything user specific away from the project and into the user's hands.

[NickM-27]

@ACiDGRiM What you're describing is already supported https://docs.frigate.video/configuration/advanced#custom-ffmpeg-build

The issue being discussed here is more complicated given that we are talking about detection and other libraries unrelated to ffmpeg which are things that can't be built / installed statically

[NickM-27]

The community support boards documentation can be found at https://docs.frigate.video/development/contributing-boards

this would be a good candidate for that