Coral edge tpu bindings for tf.js

[oveddan]

I'm not sure if this is possible, but:

the current starter code for the Coral edge board on Raspberry Pi contains python bindings for the c++ edge tpu engine. Would it be possible to create tf.js node bindings as well? It would be awesome to be able to use javascript for this, especially because of its non-blocking nature, and types provided by typescript.

[ericpelland]

I would love to see this as well!! Please and thank you!

[chenqing]

+1

[larsgk]

Another option could be to make e.g. the accelerator work directly with a web application (e.g. using WebUSB)

[dsmilkov]

Thanks for filling this. Unfortunately, we don't have the bandwidth to do this. Also, I think this would be best as an external project and the TF.js team can help provide guidance and feedback.

[EddieOne]

Using the Coral TPU is shocking fast. Would be amazing to get the Coral team to make a tfjs module. For me at least, I want to do scheduling in Node as it works with the pi camera and motor controllers. We are not too far off from having a computer that can plug itself in.

[EddieOne]

Just to update everyone

We have already received this feature request and our engineering team is analysing this. However, We don't have this on our roadmap as of now.

[bhack]

This could be interesting in the roadmap to run out of the browser: https://github.com/tensorflow/mlir/issues/140

[larsgk]

I am wondering - is there a reason for the .so files to only be closed source? If Coral were to open it, others could add support

[EddieOne]

Yeah I am disappointed with the Google Coral implementation. Maybe a better setup in my case is to use something like the Nvidia Jetson. Can someone confirm TFJS works with GPUs in NodeJs?

[moeiscool]

@EddieOne apparently tfjs works with CUDA GPUs https://github.com/tensorflow/tfjs/tree/master/tfjs-node. im not sure about the browser version

I really want to get a Coral though, but until it has Node.js support its kind of useless to me

[EddieOne]

For evaluation, it's like having a big GPU. Easily 30 frames per second. Yet, it only takes a tiny amount of power. Can run off batteries for hours with no motors running. In that way it is amazing. A lot of work if you need to do things in NodeJs + Python

[EddieOne]

Hello everyone, people are starting to notice the power of TFJS. If you want Coral support, please do this 5 min survey! https://t.co/RdQ9YreToH?amp=1

[akosb]

Please see latest WebCoral demo

[pyu10055]

To follow up, we are working with Google Coral team to provide coral-usb support on the web, stay tuned.

[larsgk]

This is awesome! - especially interesting if it works from an Android phone too :)

[ned-kelly]

Does anyone know if plans to integrate with tfjs and coral will happen any time soon?

Based on @akosb's link to the Web Demo I'm hopeful it won't be too hard to get coral support in tfjs? - Even if it's just node.js server side to begin - This is where i'm really hoping to use the library.

Cheers,

[pereibex]

+1

coral in the tfjs-node and coral in tensorflow-serving please!

[bartbutenaers]

Hi @pyu10055, @jinjingforever, Would be nice if you could share some information about the status of this feature. We are using Tensorflow.js extensively for video surveillance in our open-source commity, in the Node-RED NodeJs IOT application. If a Javascript-only solution for Coral tpu's would become available, we could offer free professional video surveillance for less technically skilled users. Thanks !!! Bart

[jasonmayes]

Hello all. I see a lot of people still interested in this and am curious to gather more information. For those of you that see this would you be able to provide more details for:

1) The target platforms you want to use this on? Is it RaspberryPi or a mobile phone or something else?

2) Is a Node.js solution just as useful or do you need browser usage explicitly? My reasoning here is that right now only Chrome (and related Chromium browsers) support this WebUSB standard but with Node.js it may be easier to interface with USB directly (just thinking out the box here). Not to mention there are issues on MacOS / Windows with the emerging WebUSB interface as detailed on this repo: https://github.com/google-coral/webcoral

3) If we had a SIG subgroup that focuses on the development of this sort of feature, who is interested in being part of that working group so we can get an idea of who may be able to help lead that effort from the external community too.

Thank you!

[pyu10055]

As @jasonmayes mentioned, the webUSB support is still limited to Chrome (requires manual firmware upload the usb connection/reset), which is not very user friendly right now. But Coral does work with this setup. The alternative is to compile TFLite + Coral runtime natively for raspberryPI and provide node.js (NAPI) binding. This way the usb initialization is automatic and transparent. We can provide the same inference API as TFJS-TFLite. cc @jinjingforever

[bartbutenaers]

Hi @jasonmayes, Thanks for the response. I will try to answer your questions as good as possible:

1. There was a community survey recently for our Node-RED application, and there you can see that Raspberry is used mostly by our users:

   

2. Node-RED runs on NodeJs, so we don't need the browser support.

3. I have asked our community whether somebody (with time and knowledge) could cooperate with your team. Hopefully somebody responds ...

@pyu10055: yes indeed that would be an alternative solution. But at the moment even our less technically skilled users can make use of Tensorflow.js, because it is installed automatically via dependencies on our application. It would be very nice if Tensorflow.js would support hardware acceleration, to make sure we keep this kind of features available to "all" kind of users. We even haven 70 year old electricity hobbyists, that are so grateful that even they can work with AI at the moment, thanks to the Tensorflow team ;-)

Thanks for your response and understanding !! Bart

[pyu10055]

@bartbutenaers Thank you for the kind words, we will definitely put coral node support on our roadmap, will update this thread when that is finalized.

[ned-kelly]

Hi @jasonmayes for me at least I'm mostly interested in running this server side (but still with lightweight) applications such as AWS Lambda functions (so x86_64) and on my IoT devices (which are arm v7 and armv8 devices) - Specifically we're using the Tinkerboard Edge T (https://tinker-board.asus.com/product/tinker-edge-t.html) which has the same built-in Edge TPU as the Google Coral SBC. Currently we're using the Python Coral libs, but i'd love to change it over to tfjs.

I think as a initial release, server side - linux based (arm and x86) support for Coral would be a good (and more practical) first step before integrating client side in Chrome - Chances are if the user has the ability to plug a coral USB device into the computer, they also have the ability to run some sort of app on the desktop which can interface directly with the TPU and the browser via a local API.

[jasonmayes]

Thank you both for your informative replies. I shall be sure to pass on details to the wider team. For other folk tracking this bug please also feel free to share more details for your intended use cases. From initial replies here and from others I have received via social media too, Node support seems to be the lower hanging fruit for now for the USB side of things at least.

It also seems some folk are using the USB dongle while others have specific boards with the TPU onboard which I guess is not via USB protocol and would need some other form of communication with it?

I think making custom drivers for every dev kit out there is not within scope as we simply do not have enough folk to support that, so maybe existing users could share some more light on how Node would be able to communicate to these TPUs that are "On board" - does the Linux distro on that device already see the TPU and able to use it - eg could one write a wrapper around whatever program currently exists to expose it to node? And would that program be different per device or universal?

Thanks.

[ned-kelly]

Hi @jasonmayes - libedgetpu.so (which I believe is what we'd need bindings for) actually interfaces directly with both the coral USB and the PCIe and when you initiate it you just pass through which device you want to use in the options - and is what the linux user usually installs (so perhaps this is the dependency you're looking for)?

This actually allows you to use multiple TPU's (either as PCIe cards which you can purchase from ASUS - or as "Mini PCIe Modules" from the Coral team) or the lightweight dev board, or even multiple USB devices (if you have a fast enough Bus)

Here's an example of how it's done currently using the pycoral api: https://coral.ai/docs/edgetpu/multiple-edgetpu/#using-the-tensorflow-lite-python-api

Unfortunately I don't know that much about developing the bindings needed for tfjs in Node, however i'd be happy to organise getting your team a sample of the Tinker Edge T from ASUS if you'd like? - I'm assuming you'd be able to get your hands on a Coral USB adapter internally also if required?

Let me know if there's anything I can do to help here Cheers,

[jasonmayes]

Thank you @ned-kelly.

I shall hand over to @pyu10055 who leads Software Engineering on the TFJS project at Google to see if he has any further questions. @pyu10055 I think it would make sense to support if it is the same driver that speaks to pcie + usb and allows user to simply specifiy which connected TPU to use at runtime assuming that is just as easy to do. We may want to test USB dongle on say Raspberry Pi, and get one of these Asus boards which has TPU on device to test both work?

[bartbutenaers]

Hi @pyu10055, I was wondering if there is any update available about this feature? Seems to be pretty difficult to order a coral tpu usb stick, because they are worldwide sold out. However last night I managed to order one. So I am going to try to get it running somehow from NodeJs on my Raspberry Pi 4. But it would be nice if this would work out-of-the-box via npm dependencies in the future, without too much of manual setup steps. Thanks and hope you have a nice summer!! Bart

[pyu10055]

@bartbutenaers We have release the coral support for node in our tfjs-sig repo, and you can follow the colab here cc @mattsoulanille

[bartbutenaers]

@pyu10055, Thanks for the pointers, and all the hard work you people are doing!!!