Tracking Issue: ESP32C3 Wifi Support

[jamesmunns]

We would like to use an ESP32C3 as a wifi coprocessor. This is to work around the fact that the exisitng Realtek (or similar) wifi chips that are available on off-the-shelf boards have very little documentation outside of the Linux Kernel itself, and even that entails roughly 250kloc for a single chip, or 400kloc for a family of chips.

There are a couple major ways to go about this:

• Use the existing esp-hosted firmware
• Write our own firmware for the ESP32C3

The boring option: use esp-hosted

There is an existing esp-hosted firmware, and embassy has support for it.

This is likely to work reasonably well.

The Cool™️ Option: write our own firmware

The other option is to port a stripped down version of mnemos to the ESP32C3, and have it act as a peer to the D1. This will allow us to experiment with some of the discussed-but-never-implemented ideas around inter-system messaging, similar to erlang's or plan9's distributed service capabilities.

This will likely rely on:

• Using embassy's HAL to implement low level drivers
• Using esp-wifi for managing the wifi process

The Plan

In #189 I added a development board with room for an RP2040 and an ESP32C3 XIAO development board. The goal here is to bring up the ESP32C3, then use the RP2040 as a "USB-to-SPI" adapter, so that we can then write the network service in melpo on the desktop, rather than going straight to the D1 development.

We'll probably go with the Cool Option unless we find it sucks.

Open Questions

How "smart" do we want the ESP32C3's network role to be?

In embassy-net, the ESP32C3 has a full network connection, handling DHCP and TCP etc directly on the device. We could keep this behavior, and have the D1 fully offload the network stack to the ESP32C3.

• Perceived Pros (of "smart c3"):
  • Fewer moving/novel parts, afaik existing code for esp-wifi is largely designed assuming you will be "consuming" the network connection locally. It's probably possible to then just proxy TCP streams and UDP frames to the D1 host.
  • Having the ability to offload means we could potentially shut down or heavily sleep the main D1 SoC, while having a maintained wifi connection. Future chips like the ESP-C5 or C6 I believe also support lower power connection modes (such as "target time to wake") which would make this even more energy economical
• Perceived Cons (of "smart c3"):
  • It is unclear the amount of CPU + RAM resources necessary to support complex networking. The CPU difference (160MHz C3, 1.008GHz D1), and RAM difference (400KiB C3, 512-1024 MiB D1) might make it appealing to keep the smarts on the bigger chip
  • It is unclear how to expose the wifi chip similarly to a low level MAC/PHY - so we can have the real network stack (likely smoltcp) running on the D1
  • It is unclear how "dynamic" we can be with opening/closing network sockets, or how many sockets we can support simultaneously, from a resource perspective

It's possible we may end up wanting to support both options, either for different use cases, or for different modes (e.g. "dumb c3" for high perf, "smart c3" for sleepy d1).

Next Steps

• [x] Port mnemos to the ESP32C3
• [ ] Bring up basic wifi capabilities (local only) on the mnemos+esp32c3
• [ ] Write a USB-UART mnemos driver to connect the ESP32C3 to Melpomene for simulator development
• [ ] Begin building out the "network coprocessor" interface, make decisions:
  • [ ] How smart should the C3 be?
  • [ ] What should the inter-system messaging link look like?
  • [ ] How do we expose this capability on the melpo host?
• [ ] Once we have a basic working network stack, bring up the RP2040 to develop the SPI interface, and proxy comms via RP2040 USB-to-SPI, instead of directly to the ESP32C3 USB-UART
  • NOTE: Since we're still capped by 12mbps USB-FS speeds, it's unlikely this will boost our total throughput, but in later board revisions we will want to turn the SPI speed up as fast as we can go (40MHz if we use FSPI pins afaict), which will be faster.
  • We are also limited to an upper speed of 26MHz over SPI with the XIAO boards, as we cannot use the FSPI pins, which are only routed to the JTAG pogo pins, and not the XIAO headers.
• [ ] Figure out a way to "perf/stress test" the SPI link, to ensure the ESP32C3 core can keep up with higher SPI speeds.

[hawkw]

Thanks for writing all of this down, this is great! I wonder if we should eventually turn this issue into an RFC doc...

One little nit: this is under "Perceived cons (of "smart c3"):"

It is unclear how to expose the wifi chip similarly to a low level MAC/PHY - so we can have the real network stack (likely smoltcp) running on the D1

Isn't this a con of the "dumb C3" approach? The "smart C3" design would have the advantage of not needing to think about exposing link-layer details over SPI and would mean the D1 <-> C3 interface only has to be aware of session-layer primitives.

[hawkw]

Out of curiosity (and I can look into this on my own): what layer does esp-hosted operate at? That might be useful prior art for deciding how much Protocol to put on the C3 and how much to run on the host...

[jamesmunns]

@hawkw I think it uses esp-hosted-fg, which means it is talking in 802.3 frames.

[hawkw]

Other thoughts: if TCP is implemented on the C3, its 400KB of RAM is probably going to be the biggest limiting factor in how many sockets we can have open at once.

How limiting it is will depend heavily on how big our tx/rx buffers are (and also on other per-socket overheads). With 4KB buffers (which is pretty small for TCP), that's 8 KB/socket...padding that by another 2 KB for misc protocol state machine variables, that gives us 10 KB/sock.

If we assume we are using about half our RAM for the mnemOS kernel, the wifi driver, SPI target driver, and misc smoltcp overhead, that gives us 200 KB of 10 KB socks, for a max of ~20 socks.

If we want bigger tx/rx bufs, we end up with substantially fewer concurrent sockets.

[hawkw]

Another potential question is whether we also want to expose the C3's Bluetooth Low Energy to the D1 in the future. IMO this would potentially be cool but only worth thinking about with after we get WiFi working. But, it might influence how smart we want the C3 to be, if we want to save some of our limited RAM for eventually running a BLE driver stack as well.

[hawkw]

How limiting it is will depend heavily on how big our tx/rx buffers are (and also on other per-socket overheads). With 4KB buffers (which is pretty small for TCP), that's 8 KB/socket...padding that by another 2 KB for misc protocol state machine variables, that gives us 10 KB/sock.

If we assume we are using about half our RAM for the mnemOS kernel, the wifi driver, SPI target driver, and misc smoltcp overhead, that gives us 200 KB of 10 KB socks, for a max of ~20 socks.

On the other hand, if we're able to immediately blast each TCP packet to the C3 over SPI, we probably don't need to keep TX/RX buffers for every socket live in RAM at all times. So, potentially, we would only need to buffer any data that's just been received and is waiting to be sent to the D1 over SPI, and data that's been received from the CPU and waiting to be sent over WiFi. So, we may need a lot less buffering, depending on what we can do with the smoltcp interface...

[jamesmunns]

@hawkw from chats with dirbaio, I'm leaning towards making the esp32c3 fairly "dumb", probably operating at a similar level to esp-hosted, and just forwarding 802.3 frames over the SPI interface.

In the future, we might be able to have an optional service that "takes over" the stream when the D1 is sleeping, basically just consuming/feeding the same frame stream that would normally go out over SPI.

[hawkw]

This seems like a good plan for now! A handoff mode for low power wake-on-LAN would be fun eventually but I agree that it's probably best to put most of the TCP layer work on the CPU.

[hawkw]

re, smartness: another constraint that's worth considering is that the esp32c3 has basically no heap...

[LegNeato]

Will this only support the coprocessor use-case or will it also support being the main/only proccessor? I have been playing around with https://shop.m5stack.com/products/m5stamp-c3-mate-with-pin-headers and wondering if this work will enable installing mnemos on it 🤔

[hawkw]

@LegNeato:

Will this only support the coprocessor use-case or will it also support being the main/only proccessor? I have been playing around with https://shop.m5stack.com/products/m5stamp-c3-mate-with-pin-headers and wondering if this work will enable installing mnemos on it 🤔

The goal is for the WiFi coprocessor firmware to be based on a MnemOS port for the ESP32-C3. We already have a minimal MnemOS core for the C3 brought up, here: https://github.com/tosc-rs/mnemos/tree/main/platforms/esp32c3-buddy

So, you could certainly write your own ESP32-C3 application using MnemOS by depending on that crate and writing your own main.rs, similar to the existing bin targets here. However, it's worth noting that we don't currently have MnemOS-style drivers for most of the ESP32-C3's peripherals --- basically all we have working right now is sending MnemOS' tracing protocol over the C3's USB serial device. If you need more than that, well, we would love your help implementing MnemOS drivers for other C3 peripherals. :)

As we start implementing the WiFi-coprocessor-specific firmware, we'll probably want to separate that out from the core MnemOS platform implementation, so that the WiFi Buddy is a separate crate that depends on the general purpose C3 core crate. That way, you can write MnemOS-on-C3 apps without having to get all of our WiFi Buddy stuff.

[LegNeato]

Ok, thanks! I've never written a driver before but I am keen to try. I'll hopefully have some time in the next couple of weeks and reach out on discord.