search

Eideticom / eid-hermes

An open-source eBPF accelerator that can run on QEMU and AWS F1 instances
10 stars 2 forks source link

Storing and Handling Command responses #16

Closed iomartin closed 3 years ago

iomartin commented 3 years ago

The theory of operation currently states that command responses are written by the device into BAR0 registers. However, there are a couple of open questions:

  1. Given that the spec allows for multiple eBPF engines and multiple XDMA channels, do we want just one set of command response registers or multiple? Multiple might make more sense long term and allow for concurrent users but poses some challenges.
  2. How can the user get the command response back? I see a few options:
    • Reading from /dev/hermesX triggers a read on BAR0 which returns the command response of the last command. This is the simplest approach but doesn't really work if we have multiple command response register sets or multiple users.
    • Sending an ioctl to /dev/hermesX allows us to pass in the command identifier as an argument. The driver can keep a map of CID --> command response and return the right values. This can work with multiple users if the driver keeps a map per PID or includes the PID in the map key.
    • Something else?
lsgunth commented 3 years ago

  1. None of this is ideal... hacking together the XDMA stuff with our registers. I suspect the best we can do is just have one set of registers and eBPF engine per XDMA channel. The kernel will need to reserve an XDMA channel or eBPF engine when it starts an action.

  2. No. The entire job should be initiated with one ioctl. The ioctl shouldn't return until the hardware has completed everything. Synchronous is fine seeing everything else is synchronous. Userspace can simply have one thread per channel/engine for parallelism.

iomartin commented 3 years ago

  1. I don't know at this point if we can enforce one eBPF engine per XDMA channel. It's of course easy to add it to the QEMU device but we don't know yet if it will fit in the FPGA. But we can have one set of registers for each eBPF engine + one for each XDMA channel.

  2. Yes, that is a good idea.

lsgunth commented 3 years ago

No use having more XDMA channels than eBPF engines and vice versa. But yes one set of registers per engine.