search

mcusim / freebsd-src

sys/dev/dpaa2 drivers work-in-progress
https://www.FreeBSD.org/
Other
4 stars 3 forks source link

MAC filter failures #14

Closed bzfbd closed 1 year ago

bzfbd commented 2 years ago 
dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3
dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3
dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3
dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3
dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3
dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3
dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3
dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3
dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3
dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3

Just got this during bootup. Not seen it before.

bzfbd commented 1 year ago

While talking I realized that on the Ten64 I see 7 of them, all on dpaa2_ni0; could it be that we try to do something on the N=0 interface for all of them instead of N=i?

dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3 dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3 dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3 dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3 dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3 dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3 dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3 Starting Network: lo0 dpni0 dpni1 dpni2 dpni3 dpni4 dpni5 dpni6 dpni7 dpni8 dpni9.

dsalychev commented 1 year ago

hm, I'll have to check on my own first

bzfbd commented 1 year ago

That seems non-sensical; we only call the ioctl on the interface which is UP so they likely are correct calls... but looking at the call path quickly I cannot spot where the 3 as error value comes from either...

dsalychev commented 1 year ago

@bzfbd Do you see the same errors on CURRENT? I've tried to netboot 17c8213559cd and got only:

# dmesg | grep dpaa2 | grep fail
dpaa2_mcp27: dpaa2_mcp_attach: failed to reset DPMCP: id=1, error=6
dpaa2_mcp27: dpaa2_mcp_attach: failed to allocate resources
dpaa2_mcp27: dpaa2_mcp_attach: failed to allocate resources
bzfbd commented 1 year ago

Yes, current and current + your latest patch both show the same problem on the ten64 for me. I should go and see on the honeycomb?

dsalychev commented 1 year ago

I haven't ever seen those errors on HoneyComb. I down-clocked my Ten64 to 1.2 GHz sometime ago:

U-Boot 2020.07-rc1-g2c6f1f4e (Jun 21 2022 - 11:26:02 +0000)

SoC:  LS1088AE Rev1.0 (0x87030010)
Clock Configuration:
       CPU0(A53):1200 MHz  CPU1(A53):1200 MHz  CPU2(A53):1200 MHz  
       CPU3(A53):1200 MHz  CPU4(A53):1200 MHz  CPU5(A53):1200 MHz  
       CPU6(A53):1200 MHz  CPU7(A53):1200 MHz  
       Bus:      500  MHz  DDR:      1600 MT/s
Reset Configuration Word (RCW):
       00000000: 30004014 00000030 00000000 00000000
       00000010: 00000000 000a0000 00300000 00000000
       00000020: 010011a0 00002580 00000000 00000000
       00000030: 013fe60a 00000000 00002c03 00000000
       00000040: 00000000 00000000 00000000 00000000
       00000050: 00000000 00000000 00000000 00000000
       00000060: 00000000 00000000 00000089 000009e7
       00000070: 44110000 0d007755
DRAM:  7.9 GiB
DDR    7.9 GiB (DDR4, 64-bit, CL=11, ECC on)
    7.4 GiB available for userspace

It might be a result of a race condition somewhere in the DPAA2 drivers, I guess.

bzfbd commented 1 year ago

Looks like the is a CLOSE happening somewhere? The XXX-BZ is printed in the error cases at the end of dpaa2_rc_exec_cmd():

dpaa2_ni0: dpaa2_ni_ioctl: cmd ADDMULTI ifp dpni0 dpaa2_ni0: dpaa2_ni_ioctl: cmd DELMULTI ifp dpni0 XXX-BZ dpaa2_rc_exec_cmd:3253 mcp 0xffffa000001187c0 cmd 0xffffa00000118e00 cmdid 0x2281 [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ] dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: cmd ADDMULTI ifp dpni0 XXX-BZ dpaa2_rc_exec_cmd:3253 mcp 0xffffa000001187c0 cmd 0xffffa00000118e00 cmdid 0x2281 [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ] dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: cmd ADDMULTI ifp dpni0 XXX-BZ dpaa2_rc_exec_cmd:3253 mcp 0xffffa000001187c0 cmd 0xffffa00000118e00 cmdid 0x2281 [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ] dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: cmd ADDMULTI ifp dpni0 XXX-BZ dpaa2_rc_exec_cmd:3253 mcp 0xffffa000001187c0 cmd 0xffffa00000118e00 cmdid 0x2281 [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ] dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: cmd ADDMULTI ifp dpni0 XXX-BZ dpaa2_rc_exec_cmd:3253 mcp 0xffffa000001187c0 cmd 0xffffa00000118e00 cmdid 0x2281 [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ] dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: cmd ADDMULTI ifp dpni0 XXX-BZ dpaa2_rc_exec_cmd:3253 mcp 0xffffa000001187c0 cmd 0xffffa00000118e00 cmdid 0x2281 [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ] dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: cmd ADDMULTI ifp dpni0 XXX-BZ dpaa2_rc_exec_cmd:3253 mcp 0xffffa000001187c0 cmd 0xffffa00000118e00 cmdid 0x2281 [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ] dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3 dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3

bzfbd commented 1 year ago

So what I noticed is that we (re-)use sc->cmd a lot but in a lot of places not under lock. Why is it that we keep using a single cmd storage rather than something on the stack?

bzfbd commented 1 year ago

Next bit:

c is a copy of the command before send off; cmd is what comes back ... These headers don't match up...

lo0: link state changed to UP
XXX-BZ dpaa2_rc_exec_cmd:3257 mcp 0xffffa000001187c0 cmd 0xffff00004039b798 cmdid 0x2281 c [ header: 0x2281020d000100d0 0: 0x2 1: 0 2: 0 3: 0 4: 0 5: 0 6: 0 ] cmd [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ]
dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3
dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3
XXX-BZ dpaa2_rc_exec_cmd:3257 mcp 0xffffa000001187c0 cmd 0xffff000132dd1298 cmdid 0x2281 c [ header: 0x2281020d000100d0 0: 0x2 1: 0 2: 0 3: 0 4: 0 5: 0 6: 0 ] cmd [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ]
dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3
dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3
XXX-BZ dpaa2_rc_exec_cmd:3257 mcp 0xffffa000001187c0 cmd 0xffff000132dd0f28 cmdid 0x2281 c [ header: 0x2281020d000100d0 0: 0x2 1: 0 2: 0 3: 0 4: 0 5: 0 6: 0 ] cmd [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ]
dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3
dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3
XXX-BZ dpaa2_rc_exec_cmd:3257 mcp 0xffffa000001187c0 cmd 0xffff000132dd0f28 cmdid 0x2281 c [ header: 0x2281020d000100d0 0: 0x2 1: 0 2: 0 3: 0 4: 0 5: 0 6: 0 ] cmd [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ]
dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3
dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3
XXX-BZ dpaa2_rc_exec_cmd:3257 mcp 0xffffa000001187c0 cmd 0xffff000132dd0f28 cmdid 0x2281 c [ header: 0x2281020d000100d0 0: 0x2 1: 0 2: 0 3: 0 4: 0 5: 0 6: 0 ] cmd [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ]
dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3
dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3
XXX-BZ dpaa2_rc_exec_cmd:3257 mcp 0xffffa000001187c0 cmd 0xffff000132dd0ee8 cmdid 0x2281 c [ header: 0x2281020d000100d0 0: 0x2 1: 0 2: 0 3: 0 4: 0 5: 0 6: 0 ] cmd [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ]
dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3
dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3
XXX-BZ dpaa2_rc_exec_cmd:3257 mcp 0xffffa000001187c0 cmd 0xffff000132dd0f28 cmdid 0x2281 c [ header: 0x2281020d000100d0 0: 0x2 1: 0 2: 0 3: 0 4: 0 5: 0 6: 0 ] cmd [ header: 0x8001026d000300d0 0: 0x5 1: 0x3e8 2: 0x3 3: 0 4: 0 5: 0 6: 0 ]
dpaa2_ni0: dpaa2_ni_update_mac_filters: failed to clear multicast MAC filters: error=3
dpaa2_ni0: dpaa2_ni_ioctl: failed to update MAC filters: error=3
dsalychev commented 1 year ago

I wanted to avoid struct dpaa2_cmd allocation/de-allocate wherever it was needed and remove calls to DPAA2_CMD_xx_OPEN() in dpaa2_ni.c. Hmm, I considered only simple use cases (like dpaa2_io.c or dpaa2_bp.c) only where the pre-allocated command is used in attach/detach function.

Anyway, the best way to solve it would be to have struct dpaa2_cmd allocated on a stack and initialized with dpaa2_mcp_init_command() without malloc, I think.

dsalychev commented 1 year ago

@bzfbd https://github.com/mcusim/freebsd-src/commit/0746e1fe33f129706efd64e8f3fcec4887079000 doesn't fail to configure MAC filters for me on Ten64 with default firmware (at 1600 MHz). EDIT: Kernel still panics under network stress test with iperf3 though.

bzfbd commented 1 year ago

I think this was sorted with one of the commits to main [1]; if I see it again I'll re-open otherwise.

[1] https://cgit.freebsd.org/src/commit/sys/dev/dpaa2?id=4cd966142822ce24c12751c863a073a8b7cb9c14