Coral Support

[dipseth]

Is there anyway to run this using a CoraTel USB Accelerator?

[stlhood]

This is an interesting device, but my understanding is that it has very little onboard memory, which means that it's unlikely to be able to run even the smaller LLMs. Apparently it also lacks several key operations needed to run these models. Some further reading:

• https://github.com/google-coral/edgetpu/issues/668
• https://www.reddit.com/r/LocalLLaMA/comments/13n613w/why_cant_we_use_edge_tpus_like_coral_ai_with/

[jart]

Can it at least run Rocket 3b? Since Rocket runs great for me on a similarly priced $60 raspberry pi 4.

[TomatoCo]

The coral TPUs have only 8 megabytes of memory. Their main advantage is that they do about 4 trillion operations per second while drawing one watt.