search

brilliantlabsAR / frame-codebase

The complete codebase for Frame
https://www.brilliant.xyz
Other
207 stars 41 forks source link

[feature request] Video input over Bluetooth #152

Closed josuah closed 4 months ago

josuah commented 6 months ago

This is a discussion to explore all the ways to implement video transfer over Bluetooth towards Frame display.

The 2 point that could throttle the transfer might be:

As well as amount of processing on the MCU and FPGA.

A few evoked solutions are:

For reference, the absolute maximum that is possible to achieve on an nRF52840 with Bluetooth 5: https://www.youtube.com/watch?v=K1ItqEZ2_tw

josuah commented 6 months ago

To illustrate what different resolution and color mode mean in practice: None of these were confirmed to be possible yet!

400x400 full RGB888: image_400_888

400x400 16 indexed colors: image_400_16i

200x200 full RGB888: image_200_888

200x200 16 indexed colors: image_200_16i

100x100 full RGB888: image_100_888

100x100 16 indexed colors: image_100_16i

josuah commented 6 months ago

Here is what could be done for the Monocle: output

Not yet tested on the hardware.

josuah commented 5 months ago

This could be achieved on hardware on the Monocle mpv-shot0008

As seen above, still limitations that depend on the complexity of the image.

rohitrathnam commented 5 months ago

Deep dive into how JPEG works: Everything You Need to Know About JPEG Tutorial Series Leaving this here for future reference

rmetchev commented 5 months ago

Git repo to with the video series, used as reference for encoder. https://github.com/dannye/jed

rohitrathnam commented 4 months ago

Quick update, the EBR/LUT resources are almost full so it's unlikely we can support both JPEG decode and encode at the same time

josuah commented 4 months ago

If two identical bitstreams can fit on the flash, this means it is possible to restart the FPGA at runtime with the alternative bitfile to cycle through features as the API uses them?

I think the LZO-compressed image was still quite large already though...

siliconwitch commented 4 months ago

Some great links here. The LZO compression is probably the best option for fast uncompressing on the device (API coming soon), but this isn't very useful for images.

Unfortunately, there aren't enough resources left for jpeg decompression on the device.

However slow frame rates could work. Remember that Frame can only show 16 colors at a time per frame. This can reduce the format down to 4bbp images. Combined with some posterization off-device and LZO, it could be possible to get a few FPS. Send the compressed bytes over the Bluetooth data channel, decompress, and print to display using frame.display.bitmap() (details coming soon once the API is done)

Keep an eye on issues: https://github.com/brilliantlabsAR/frame-codebase/issues/114 and https://github.com/brilliantlabsAR/frame-codebase/issues/81

josuah commented 4 months ago

Some feedback suggests to investigate the https://multimedia.cx/mirror/idroq.txt codec widely used on early inexpensive (read: low-resource, low-complexity codec), high volume game consoles.

https://discord.com/channels/963222352534048818/984966420603482182/1240281670939050075 https://github.com/id-Software/Quake-III-Arena/blob/master/code/client/cl_cin.c