lukego / easynic

EasyNIC: an easy-to-use host interface for network cards
43 stars 0 forks source link

Hardware for 10G EasyNIC #15

Open lukego opened 5 years ago

lukego commented 5 years ago

Suppose we wanted to build a 10G EasyNIC. Is there suitable hardware available?

Surprisingly to me, the answer seems to be yes. Here is a suggestion from a twitter conversation with @daveshah1 and others:

EasyNIC 10G would be:

The FPGA would use programmable logic to implement the PCIe endpoint, the Ethernet MAC, and the EasyNIC driver interface. The I/O interface from FPGA to PCIe would be the 4 x 5Gbps SERDES on the ECP5-5G. The I/O interface from FPGA to 10G PHY would be 32x311Mbps using ordinary I/O pins on the FPGA (which is known to support twice this bitrate for DDR3 memory.)

The cost of the FPGA seems to be ~$50 and the PHY ~$20. There are no license fees for the developer tools or hardware features thanks to full ECP5 support in the Yosys open source hardware toolchain.

The NIC might be able to have multiple 10G ports each using separate silicon and connecting to a "bifurcated" PCIe slot.

This is an exciting possibility. Overall this NIC would seem comparable to the Intel 82599. This would make it a practical NIC for many serious applications.

If we decide to move forward with this approach then we could start by developing a 1G NIC using an off-the-shelf ECP5 Versa development board that costs ~$200 and includes PCIe and 10/100/1000 RJ45 connectivity.

daveshah1 commented 5 years ago

Some more thoughts on this.

lukego commented 5 years ago

@daveshah1 Great to have you here! :-)

I reckon the typical Snabb user, often working in an ISP or similar environment, would go for the 1xSFP+ option. SFP+ because networks often use fiber and 1x because people like to know the worst-case performance for their equipment and plan their networks to avoid bottlenecks. There is actually quite a bit of value in having PCIe headroom for peace of mind (another way of saying the value of the second SFP+ would be much less than the first one.)

lukego commented 5 years ago

Just to be concrete. Some NICs like the Intel 82599 have lots of PCIe headroom and use PCIe protocols efficiently. It's really hard to make an 82599 drop a packet and so when a packet gets lost you don't have to worry about PCIe being the problem.

Other NICs drop packets all the time in normal operation and make it hard to know what the safe or expected performance should be (e.g. https://github.com/snabbco/snabb/issues/1013#issuecomment-246124699). These make planning and troubleshooting much more difficult.

lukego commented 5 years ago

@daveshah1 One more idea kicked around on the Snabb slack channel is a PCIe3-to-PCIe2 bridge IC. Then we could have Nx10G each with a dedicated ECP5 and PHY all sharing (say) 64Gbps of PCIe 3.0 x8 bandwidth.

Maybe for a future design iteration...

MegabytePhreak commented 5 years ago

Saw this on twitter, one thing I noticed: Use caution with the line side interface on the vsc8486. XFI is for XFP modules, not SFP+, which use SFI. The difference is in how much signal conditioning is done on the module, more for XFI, less for SFI. So SFI is more challenging for the PHY. The VSC8486-4 datasheet makes some noises about supporting SFP+, but in a way that doesn't give me 100% confidence. This can be particularly important with twinax cables.

mslavescu commented 5 years ago

@lukego, @daveshah1 would be possible to build a bottom board, with 2.5/5/10G EasyNIC, for the Lattice Embedded Vision Development Kit? https://www.digikey.ca/en/product-highlight/l/lattice/lf-evdk1-evn-fpga-modular-video-platform

Would this provide better (lower/consistent) latency than the USB3 from this expansion board for the kit: https://www.digikey.ca/product-detail/en/lattice-semiconductor-corporation/USB3-VIP-EVN/220-2159-ND/8619596

or the USB3 on this board: https://www.latticesemi.com/Products/DevelopmentBoardsAndKits/LatticeUSB3VideoBridgeDevelopmentKit

Or would be possible to connect the top dual camera part (from Lattice vision kit) directly to an Avnet Ultra96 FPGA board though the high-speed interface like in this expansion board: https://www.96boards.org/product/mipiadapter/

I'm looking for affordable ways to build smart (stereo/multi) cameras for robotics and self driving cars, as part of http://ossdc.org/.