search

mcusim / freebsd-src

sys/dev/dpaa2 drivers work-in-progress
https://www.FreeBSD.org/
Other
4 stars 3 forks source link

ten64: Slow or stall enumerating DPNIs on boot #13

Closed mcbridematt closed 2 years ago

mcbridematt commented 2 years ago

I tried to run the latest code but am stuck at this problem.

At boot it there is a very long pause (or stall) while enumerating the DPNIs:

[2022-06-25 22:04:26.817] dpaa2_rc0: dpaa2_rc_discover: failed to get object: idx=30, error=253
[2022-06-25 22:04:26.832] dpaa2_ni0: <DPAA2 Network Interface> dpio (id=5-7,0-4) dpbp (id=0) dpcon (id=21-28) dpmcp (id=4) at dpni (id=9) on dpaa2_rc0
[2022-06-25 22:04:26.870] dpaa2_ni0: connected to dpmac (id=7)
[2022-06-25 22:04:26.893] dpaa2_ni0: connected DPMAC is in FIXED mode
[2022-06-25 22:04:26.893] dpaa2_ni0: channels=8
[2022-06-25 22:04:27.612] dpni0: Ethernet address: 00:0a:fa:24:24:fd
[2022-06-25 22:04:27.612] dpaa2_ni1: <DPAA2 Network Interface> dpio (id=7,6,5,4,3,2,1,0) dpbp (id=1) dpcon (id=29-30,6-11) dpmcp (id=3) at dpni (id=8) on dpaa2_rc0
[2022-06-25 22:04:27.626] dpaa2_ni1: connected to dpmac (id=8)
[2022-06-25 22:04:27.649] dpaa2_ni1: connected DPMAC is in FIXED mode
[2022-06-25 22:04:27.649] dpaa2_ni1: channels=8
[2022-06-25 22:04:28.349] dpni1: Ethernet address: 00:0a:fa:24:24:fe
[2022-06-25 22:04:28.349] dpaa2_ni2: <DPAA2 Network Interface> dpio (id=7,6,5,4,3,2,1,0) dpbp (id=2) dpcon (id=12-19) dpmcp (id=2) at dpni (id=7) on dpaa2_rc0
[2022-06-25 22:04:28.363] dpaa2_ni2: connected to dpmac (id=9)
[2022-06-25 22:04:28.386] dpaa2_ni2: connected DPMAC is in FIXED mode
[2022-06-25 22:04:28.386] dpaa2_ni2: channels=8
[2022-06-25 22:04:29.058] dpni2: Ethernet address: 00:0a:fa:24:24:ff
[2022-06-25 22:04:29.086] dpaa2_ni3: <DPAA2 Network Interface> dpio (id=7,6,5,4,3,2,1,0) dpbp (id=3) dpcon (id=20,0-5,70) dpmcp (id=1) at dpni (id=6) on dpaa2_rc0
[2022-06-25 22:04:29.101] dpaa2_ni3: connected to dpmac (id=10)
[2022-06-25 22:04:29.123] dpaa2_ni3: connected DPMAC is in FIXED mode
[2022-06-25 22:04:29.123] dpaa2_ni3: channels=8
[2022-06-25 22:04:29.795] dpni3: Ethernet address: 00:0a:fa:24:25:00
[2022-06-25 22:04:29.795] dpaa2_ni4: <DPAA2 Network Interface> dpio (id=7,6,5,4,3,2,1,0) dpbp (id=4) dpcon (id=71-78) dpmcp (id=28) at dpni (id=5) on dpaa2_rc0
[2022-06-25 22:04:29.837] dpaa2_ni4: connected to dpmac (id=3)
[2022-06-25 22:04:29.837] dpaa2_ni4: connected DPMAC is in FIXED mode
[2022-06-25 22:04:29.859] dpaa2_ni4: channels=8
[2022-06-25 22:04:30.638] dpni4: Ethernet address: 00:0a:fa:24:25:01
[2022-06-25 22:04:30.638] dpaa2_ni5: <DPAA2 Network Interface> dpio (id=7,6,5,4,3,2,1,0) dpbp (id=5) dpcon (id=79,55-60,62) dpmcp (id=27) at dpni (id=4) on dpaa2_rc0
[2022-06-25 22:04:30.676] dpaa2_ni5: connected to dpmac (id=4)
[2022-06-25 22:04:30.676] dpaa2_ni5: connected DPMAC is in FIXED mode
[2022-06-25 22:04:30.676] dpaa2_ni5: channels=8
[2022-06-25 22:04:32.001] dpni5: Ethernet address: 00:0a:fa:24:25:02
[2022-06-25 22:04:32.001] dpaa2_ni6: <DPAA2 Network Interface> dpio (id=7,6,5,4,3,2,1,0) dpbp (id=6) dpcon (id=61,63-69) dpmcp (id=26) at dpni (id=3) on dpaa2_rc0
[2022-06-25 22:04:32.017] dpaa2_ni6: connected to dpmac (id=5)
[2022-06-25 22:04:32.039] dpaa2_ni6: connected DPMAC is in FIXED mode
[2022-06-25 22:04:32.039] dpaa2_ni6: channels=8
[2022-06-25 22:04:34.267] dpni6: Ethernet address: 00:0a:fa:24:25:03
[2022-06-25 22:04:34.267] dpaa2_ni7: <DPAA2 Network Interface> dpio (id=7,6,5,4,3,2,1,0) dpbp (id=7) dpcon (id=40-47) dpmcp (id=25) at dpni (id=2) on dpaa2_rc0
[2022-06-25 22:04:34.283] dpaa2_ni7: connected to dpmac (id=6)
[2022-06-25 22:04:34.305] dpaa2_ni7: connected DPMAC is in FIXED mode
[2022-06-25 22:04:34.305] dpaa2_ni7: channels=8
[2022-06-25 22:04:36.359] dpni7: Ethernet address: 00:0a:fa:24:25:04
[2022-06-25 22:04:36.359] dpaa2_ni8: <DPAA2 Network Interface> dpio (id=7,6,5,4,3,2,1,0) dpbp (id=8) dpcon (id=48-54,31) dpmcp (id=24) at dpni (id=1) on dpaa2_rc0
[2022-06-25 22:04:36.366] dpaa2_ni8: connected to dpmac (id=2)
[2022-06-25 22:04:36.391] dpaa2_ni8: connected DPMAC is in FIXED mode
[2022-06-25 22:04:36.391] dpaa2_ni8: channels=8
[2022-06-25 22:05:14.196] dpni8: Ethernet address: 00:0a:fa:24:25:06
[2022-06-25 22:05:14.196] dpaa2_ni9: <DPAA2 Network Interface> dpio (id=7,6,5,4,3,2,1,0) dpbp (id=9) dpcon (id=32-39) dpmcp (id=23) at dpni (id=0) on dpaa2_rc0
[2022-06-25 22:05:14.258] dpaa2_ni9: connected to dpmac (id=1)
[2022-06-25 22:05:14.258] dpaa2_ni9: connected DPMAC is in FIXED mode
[2022-06-25 22:05:14.258] dpaa2_ni9: channels=8
[2022-06-25 22:07:09.710] dpni9: Ethernet address: 00:0a:fa:24:25:05
[2022-06-25 22:07:09.786] gpioled0: <GPIO LEDs> on ofwbus0

Note the timestamps, the last couple of DPNIs take longer to enumerate. Between dpaa2_ni8: channels=8 and dpni8: Ethernet address: 00:0a:fa:24:25:06 there is about a 40 second delay. After dpaa2_ni9: channels=8 there is a delay of two minutes.

On one system it gets stuck at dpni7 (and gets stuck there), on the other it stalls for a minute at dpni9. (Coincidentally, dpni8 and 9 are the SFP ports) I tried both MC firmware 10.20 (current Ten64 default) and 10.29.1 without any change in behaviour.

I bisected the issue to commits 19d82451a68fcd710b3c0a03b2da843b8238a407 and 846462f2d7ec1ae10fbe90da9278a06103af61a2

Commit e856e7a07323a91d532a19af831f4cde63f83cbc is the last good commit

dsalychev commented 2 years ago

@mcbridematt Lazy me decided that counting buffers taken by QBMan from the pool isn't that necessary and just allocated and released as many buffers to the pool as possible at the dpni attach at 846462f2d7ec1ae10fbe90da9278a06103af61a2... uhh.

I'll probably have to implement a mechanism which will be allocating and releasing new buffers when a threshold is crossed (similar to what's done in the Linux driver).

dsalychev commented 2 years ago

@mcbridematt Could you test with 1a7aba9f89185b0533b46ecb641560a6be3cb614?

mcbridematt commented 2 years ago

No enumeration / boot issues with https://github.com/mcusim/freebsd-src/commit/1a7aba9f89185b0533b46ecb641560a6be3cb614