brilliantlabsAR / frame-codebase

The complete codebase for Frame
https://www.brilliant.xyz
Other
207 stars 41 forks source link

[feature-request] Add some lua bindings for a few toolkits from CMSIS #134

Closed josuah closed 4 months ago

josuah commented 7 months ago

The CMSIS library already used by Frame comes with optimized functions for various operations including classical DSP and Neural Networks. This could i.e. be used as preprocessing for audio:

https://arm-software.github.io/CMSIS-NN/latest/index.html https://arm-software.github.io/CMSIS-DSP/latest/index.html

Having them in the firmware without even doing anything with them would let users experiment with their own features, i.e. low-res video or audio processing could eventually fit.

rohitrathnam commented 4 months ago

We can add math operations, but not sure how to implement the vector ones. It expects pointers, and the overhead to go in/out from lua table to C array would probably kill any gains. Any ideas?

josuah commented 4 months ago

CMSIS-NN and CMSIS-DSP have a lot of content... Maybe some very practical use-cases will enlight us about what we could try to add.

A few audio ideas:

A few image ideas:

Some might require a lot of work, a lot of resources, but hopefully some are low-hanging fruits.

josuah commented 4 months ago

CMSIS might not be a good target when the BUS is SPI... We have our answer! Thank you! :)