Embedded DepthAI + Thermal Reference Design

[Luxonis-Brandon]

Start with the why:

There are many applications for which thermal information is incredibly useful, particularly when paired with high-res RGB and depth information. Previously-unsolvable industrial, agricultural, and medical problems are much easier solved with this pairing. Examples include easier understanding machine state (i.e. does that pipe have high pressure/heat behind it... should an alarm go off if a worker approaches it).

So adding something like the MLX90640 from the Open Thermal Camera to https://github.com/luxonis/depthai-hardware/issues/10 could be incredibly useful. The pairing of the high-res RGB and depth allows augmenting the lower-resolution of the Melexis in interesting ways, particularly using ML techniques to aid in accuracy/etc.

Move to the how:

Evaluate if the board size of https://github.com/luxonis/depthai-hardware/issues/10 needs to be increased to add the MLX sensor. Connect the MLX into the Myriad directly so that ML/CV techniques can be applied directly to this sensors output (and/or fused w/ the other sensor data).

Move to the what:

Make a version of https://github.com/luxonis/depthai-hardware/issues/10 with integrated thermal sensor.

[Luxonis-Brandon]

Here's our first stab at getting thermal running:

Looking pretty decent actually.

[HannesGurschler]

Hi Brandon,

hope you still remember me :) wow this looks really great! But which Sensor do you use here the MLX ir the Flir Lepton? Because the reolution looks quite good?

You implemented the thermal sensor as the celntral camera right? So it is not possible to get the depth information of the thermal image?

It's really cool how fast you guys are! :) Can't wait to get the OAK-D into my handy.

When do you think this Prototype is available?

All the best and thank you very much, Hannes

Von meinem iPhone gesendet

Am 14.12.2020 um 17:22 schrieb Luxonis-Brandon notifications@github.com:



Here's our first stab at getting thermal running: [image]https://user-images.githubusercontent.com/32992551/102106476-dbda6a80-3ded-11eb-90c1-4c4c2c1fb2c3.png

Looking pretty decent actually.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/luxonis/depthai-hardware/issues/13#issuecomment-744551059, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQPWT7VSSCDJGYBXPPKKNLDSUY3VHANCNFSM4PHEXHAQ.

[Luxonis-Brandon]

Hi HannesGurschler!

Thanks for the kind words!

So it is this sensor:

We are investigating integration now. This module is actually USB, so when it is used with DepthAI, DepthAI will be a USB host (a functionality we don't advertize really anywhere), so then the interface to DepthAI could by Ethernet, WiFi, BT, or SPI (but not USB, as it would be consumed by DepthAI being the host). Likely prototypes for these will be in Q1 2021. And we're excited to get OAK out to everyone... these are actively being trucked to our shipping partner (floship) now, with over 920 having arrived at floship already.

We will be prototyping other thermal sensors as well.

Vielen Dank, -Brandon

[HannesGurschler]

Hi Brandon, okay sounds great. Looking forward! Best regards, Hannes

Von meinem iPhone gesendet

Am 16.12.2020 um 05:05 schrieb Luxonis-Brandon notifications@github.com:



Hi HannesGurschler!

Thanks for the kind words!

So it is this sensor: [image]https://user-images.githubusercontent.com/32992551/102303495-7f279e80-3f18-11eb-867c-760c0ed09321.png [image]https://user-images.githubusercontent.com/32992551/102303611-cf9efc00-3f18-11eb-887f-4cfc798c2bc3.png

We are investigating integration now. This module is actually USB, so when it is used with DepthAI, DepthAI will be a USB host (a functionality we don't advertize really anywhere), so then the interface to DepthAI could by Ethernet, WiFi, BT, or SPI (but not USB, as it would be consumed by DepthAI being the host). Likely prototypes for these will be in Q1 2021. And we're excited to get OAK out to everyone... these are actively being trucked to our shipping partner (floship) now, with over 920 having arrived at floship already.

We will be prototyping other thermal sensors as well.

Vielen Dank, -Brandon

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/luxonis/depthai-hardware/issues/13#issuecomment-745749782, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQPWT7VXA62UJ46NMW3EPG3SVAWW7ANCNFSM4PHEXHAQ.

[kabhari]

Hi @Luxonis-Brandon,

Is this still on the roadmap by any chance? If so, any idea of the timeline?

Thanks, K

[Erol444]

Hi @kabhari , Great news, thermal sensors are just on their way to our offices, so we can create an initial PCB design with them. So I estimate early prototypes will be available Q3, and release of, at least FFC product, in Q4 this year. Disclaimer: Timelines could change if we determine that these sensors don't fit our requirements. Thoughts? Thanks, Erik

[stephansturges]

Hi @kabhari , Great news, thermal sensors are just on their way to our offices, so we can create an initial PCB design with them. So I estimate early prototypes will be available Q3, and release of, at least FFC product, in Q4 this year. Disclaimer: Timelines could change if we determine that these sensors don't fit our requirements. Thoughts? Thanks, Erik

Very interested in these modules, please post a link here when there is a pre-order available!

[Erol444]

Feel free to signup for email notifications on Thermal Sensor:) https://share-eu1.hsforms.com/1LJhyxiHYSvqGIJ6J5uU3wwf2chh

[stephansturges]

Done, thanks :) I didn't realise you had this list.

[kabhari]

@Erol444 Thanks a lot, this is great news. We are also very interested in the early prototype and/or the FCC product so looking forward to it.

[stephansturges]

Just wanted to say this looks cool and I'm still excited to see more 😄