I210 AVB on Xilinx Zynq ARM

[dbol-elium]

Hi. We are trying to port Intel I210 based Open AVB project into Xilinx Zynq ARM based platform. It's supposed to communicate with Intel I210 Ethernet adapter card via FPGA PCIe IP core (axi_pcie).

We've got igb driver from mainline kernel (xlnx-4.0 linux kernel) working (at least in legacy mode for now).

So, now we are trying to get igb_avb kernel module (AVB-customed version of the Intel igb driver) working for our ARM based platform. We've cross-compiled igb_avb driver successfully. The module is well probed and ETH is instantiated: also using legacy interrups, failed to initiaze MSI/MSI-X interrupts (same as mainline igb driver - needs further research also for FPGA PCIe core + drivers). But unlike the standard igb driver, after bringing up the ETH looses any TX/RX packets (showing the link is up at 1000Mb/s). Diff for the igb_avb source with the standard igb driver from linux 4.0 kernel shows a plenty of changes/differences. Looking at igb_avb kcompat.h and Makefile shows some dependencies on kernel version and features, linux distribution etc - which is not present in standard igb driver from linux 4.0 kernel. So, need to investigate igb_avb source code - possibly some defines are missed or something else related to arm linux needs to be fixed and so on...

Now we have some few questions for further development within igb_avb:

1. Has anyone successfully ported and used IGB AVB driver on ARM platform? Possibly there are some known fixups/patches to get igb_avb working on ARM etc...

2. Is MSI-X compatibility mandatory for IGB AVB-customed version (AVB functionality)? I'm not sure that Xilinx PCIe is fully compliant with MSI-X. There are some collisions in Xilinx documentation for that and some explanations from Xilinx needed. So, if the MSI-X is manadatory for I210 AVB features we need to see into Xilinx PCIe first...

3. What are the other kernel / platform dependencies needed for AVB feature?

Thanks in advance.

[intel-ethernet]

I can’t speak about ARM bringup, although I don’t recall anything architecture specific in the igb_avb driver.

I’d suspect your kernel .config settings. You shouldn’t need MSI//MSI-X.

Biggest item is RSS / MQ (multi-queue) must be enabled – but I think you’d get a compile-time error if this weren’t the case.

error Must have multi-queue tx support enabled (CONFIG_NETDEVICES_MULTIQUEUE)!

Are you compiling on one system and then copying the resulting driver over to the target?

ekm

[dbol-elium]

Yes, I cross-compile via Xilinx petalinux toolchain on PC Linux. And then copy/insmod on the target Zynq ARM device.

Also just checked for CONFIG_NETDEVICES_MULTIQUEUE in target kernel config (its mainline 4.0 kernel patched by xilinx for FPGA cores support): there is nothing for CONFIG_NETDEVICES_MULTIQUEUE (i.e. is disabled) And also there is no compile-time error for this... Needs some investigation within igb_avb source code.

The reason why i'm worrying about MSI/MSI-X is the following: There is comment note "... This driver requires (4) tx-rx queues are enabled, with MSI-X. ..." inside igb_main.c. As I wrote we've reached standard igb driver from kernel working. But it works only in legacy mode for now. For some reason pci_enable_msi() and pci_enable_msix_range() fail for us...

So, not clear whether MSI/MSI-X is manadatory for I210 AVB features or not? If there is no need for MSI/MSI-X I can research to get it working also in legacy mode...

P.S. Thanks for the quick reply.

[intel-ethernet]

I only have tested/used MSI-X so I don’t know if there is a defect or not … but looking at the code, ‘legacy interrupt’ should work. But you must have MULTIQUEUE enabled or nothing (no tx, no rx) will work with the existing driver.

ekm

[dbol-elium]

Just have looked in the code within igb_main.c:

static void igb_set_interrupt_capability(struct igb_adapter _adapter, bool msix) { ... case IGB_INT_MODELEGACY: / disable advanced features and set number of queues to 1 _/ igb_reset_sriov_capability(adapter); adapter->vmdq_pools = 0; adapter->rss_queues = 1; adapter->flags |= IGB_FLAG_QUEUE_PAIRS; adapter->num_rx_queues = 1; adapter->num_tx_queues = 1; adapter->num_qvectors = 1; / Don't do anything; this is system default */ break; } }

As I understand it means that without MSI/MSI-X AVB features will be not available... Am I right?

[dbol-elium]

I've also checked for CONFIG_NETDEVICES_MULTIQUEUE - google tells that: "CONFIG_NETDEVICES_MULTIQUEUE only exists from 2.6.23 to 2.6.26. Other options enable multiqueue in 2.6.27 and newer kernels."

And also looked in the code within kcompat.h:

if ( LINUX_VERSION_CODE < KERNEL_VERSION(2,6,27) )

...

ifdef CONFIG_NETDEVICES_MULTIQUEUE

define HAVE_TX_MQ

endif

...

else /* < 2.6.27 _/

define HAVE_TX_MQ

define HAVE_NETDEV_SELECT_QUEUE

ifdef CONFIG_DEBUG_FS

define HAVE_IXGBE_DEBUG_FS

define HAVE_IGB_DEBUG_FS

endif /_ CONFIG_DEBUGFS /

endif /_ < 2.6.27 */

In our case (kernel 4.0) there is no compile-time validation for CONFIG_NETDEVICES_MULTIQUEUE. That's why there is no compile-time error for that.

But after bringing up the ETH I210 still loosed any TX/RX packets (showing the link is up at 1000Mb/s). Also ethtool -S shows TX/RX packets are available... But ping fails and ifconfig tells RX/TX: 0

So, now the question is what options should be used to enable RSS/Multiqueue in kernel 4.0? And what else needed to get it working?

Thanks in advance.

[intel-ethernet]

Your best bet is to look at the statistics registers to determine:

a) Whether you are receiving packets

b) Whether the I210 is dropping packets because it can’t DMA into buffers (no_buffer)

c) …

[dbol-elium]

Hi.

For now we've got I210 Ethernet adapter up (after some updates on Xilinx side to xlnx-4.6 linux kernel and newer FPGA PCIe core glue logic).

So, now MSI-X and 4 rx-tx queues are enabled. Standard best-effort LAN traffic via rx-tx-queue(3) seems to be working (i.e. ping, file transfers work). I've also noticed a significant number of dropped standard RX packets reported by ifconfig. And it grows approx. by 1 dropped packet every 1-2 seconds. But standard LAN traffic and AVB RX works.

But now we've met another issue within AVB stuff (igb lib - transmission):

Each transmission on tx-queue(0) through tx-queue(2) causes E1000_TXD_STAT_EC (i.e 0x00000002, Excess Collisions) status bit in TX descriptor and dropped packet.

So, what are the possible reasons of "Excess Collision" on every TX packet? I can also send the I210 register dump (the output of ethtool -d) if it will help to find the reason...

Thanks in advance and kind regards.

[intel-ethernet]

Collisions?

Are you running half-duplex ???? Maybe there is something wrong with your board design ?

ekm