search

crossroadsfpga / pigasus

100Gbps Intrusion Detection and Prevention System
671 stars 76 forks source link

Use Intel XL710 for 40GbE QSFP #18

Open lapnd opened 2 years ago

lapnd commented 2 years ago

Hi @hsadok I have single PC with both Stratix MX board and XL710 for 40GbE QSFP (both Pigasus and Pktgen will be run on the same machine. The Stratix MX and X710 NICs are connected together).

The Pigasus is starting OK. However when I start pktgen, I saw that the link is down.

/ Ports 0-1 of 2   <Main Page>  Copyright(c) <2010-2021>, Intel Corporation
  Flags:Port        : P------Sngl       :0 P------Sngl       :1
Link State          :           <--Down-->           <--Down-->     ---Total Rate---
Pkts/s Rx           :                    0                    0                    0
       Tx           :                    0                    0                    0
MBits/s Rx/Tx       :                  0/0                  0/0                  0/0
Pkts/s Rx Max       :                    0                    0                    0
       Tx Max       :                    0                    0                    0
Broadcast           :                    0                    0
Multicast           :                    0                    0
Sizes 64            :                    0                    0
      65-127        :                    0                    0
      128-255       :                    0                    0
      256-511       :                    0                    0
      512-1023      :                    0                    0
      1024-1518     :                    0                    0
Runts/Jumbos        :                  0/0                  0/0
ARP/ICMP Pkts       :                  0/0                  0/0
Errors Rx/Tx        :                  0/0                  0/0
Total Rx Pkts       :                    0                    0
      Tx Pkts       :                    0                    0
      Rx/Tx MBs     :                  0/0                  0/0
TCP Flags           :               .A....               .A....
TCP Seq/Ack         :  305419896/305419920  305419896/305419920
Pattern Type        :              abcd...              abcd...
Tx Count/% Rate     :           100000 /1%           100000 /1%
Pkt Size/Tx Burst   :            64 /  128            64 /  128
TTL/Port Src/Dest   :       64/ 1234/ 5678       64/ 1234/ 5678
Pkt Type:VLAN ID    :      IPv4 / TCP:0001      IPv4 / TCP:0001
802.1p CoS/DSCP/IPP :            0/  0/  0            0/  0/  0
VxLAN Flg/Grp/vid   :     0000/    0/    0     0000/    0/    0

This is what I have tried with pktgen

sudo cp $pigasus_rep_dir/pigasus/hardware/rtl_sim/input_gen/m10_100.pcap /dev/shm/test.pcap
sudo /opt/pktgen-dpdk/Builddir/app/pktgen -c 0x000000000000003C  -- -P -m "[3].0, [4].1"

I know that the instruction is for Mellanox 100Gbps NIC. On the quartus project, I can see Pigasus using 100G as image

I'm newbie with Pigagus and want to try to run with all I have now. Is it ok to run the test with other NICs like Intel XL710 40Gbps NIC? Thank you!

hsadok commented 2 years ago

Hi,

A Mellanox NIC is definitely not required. However, I don't think we ever tried is to connect a 40Gbp NIC to Pigasus. Not sure if this could cause an issue.

Running Pigasus and DPDK pktgen on the same machine should be okay though. We have done it before as well.

You may try connecting the cable between the two ports of your NIC just to see if the link gets up. If it doesn't, your problem is likely unrelated to Pigasus. If it does, it might be an issue related to connecting 40Gb to 100Gb (but this is conjecture).

lapnd commented 2 years ago

Hi @hsadok The link gets up when I connect two ports together

/ Ports 0-1 of 2   <Main Page>  Copyright(c) <2010-2021>, Intel Corporation
  Flags:Port        : P------Sngl       :0 P------Sngl       :1
Link State          :        <UP-40000-FD>        <UP-40000-FD>     ---Total Rate---
Pkts/s Rx           :                    0                    0                    0
       Tx           :                    0                    0                    0
MBits/s Rx/Tx       :                  0/0                  0/0                  0/0
Pkts/s Rx Max       :                    0                    0                    0

So the NICs and cables are OK. Do we have any loop back test with Pigagus when connect two ports of Stratix board together?

hsadok commented 2 years ago

Thank you for giving that a try. At least now we know that the NIC is probably working fine.

Do we have any loop back test with Pigagus when connect two ports of Stratix board together?

I don't think so -- at least not that I'm aware.

We might need thoughts from @zhipengzhaocmu here. But if you have access to a 100 Gb NIC that would be the first thing that I would try.

lapnd commented 2 years ago

Thank you @hsadok Can you please recommend the Mellanox card model ? I found some of them but I'm not sure which one is the suitable/tested model:

zhipengzhaocmu commented 2 years ago

Thanks for trying out Pigasus.

I tried to connect a 40G NIC with a MX FPGA board with 100G MAC IP in the very early stage of Pigasus project. It didn't work out of box. I believe the 100 MAC IP setting disabled the auto-negotiation/link-training by default(https://www.intel.com/content/www/us/en/docs/programmable/683430/21-1-19-3-0/parameter-editor-parameters.html) so any line rate other than 100Gbps won't work with the 100 MAC IP on the MX FPGA board. I didn't spend time to make the 40G NIC work with Pigasus. But I think it is worth a try to enable that parameter if buying a new 100Gbps NIC is not in your original plan.

Do we have any loop back test with Pigagus when connect two ports of Stratix board together?

We do have a project that performs external loopback of the MX FPGA board---connecting the two QSFP ports using a cable, one port sends some data to another port which checks the data. An external team has also tested it during their board bring-up phase. Once we find it, we will make it part of this repo to help board bring-up.

hsadok commented 2 years ago

The exact Mellanox NIC we used was: "Mellanox Technologies MT27800 Family [ConnectX-5]"

But really any 100Gb NIC should do.

lapnd commented 2 years ago

Thank you @zhipengzhaocmu and @hsadok I tried to enable auto-negotiation, but the link still down. I will get a Mellanox NIC to try.

lapnd commented 2 years ago

I found that, Stratix board also support IP core 40Gbps as https://www.intel.com/content/www/us/en/docs/programmable/683793/current/supported-ethernet-ip-cores-and-devices.html

Another option I may give a try is using this 40Gbps mode.

zhipengzhaocmu commented 2 years ago

If you are familiar with FPGAs and IP cores, then changing the Ethernet IP to 40Gbps is viable. Just a heads-up, this change could be non-trivial, including regenerating the 40G IP example design, connecting Pigasus processing logic with the 40G IP by inserting width conversion logic, updating the .qsf and .sdc files to make Quartus happy.

In the meanwhile, you can give this PR a try. It is the external loopback test, it would tell you if the FPGA board and cable are working or not. It's still under review, let me know if you face any issues.