Use DPMCPs for communicating with MC / fix VFIO guest

[mcbridematt]

Under VFIO passthrough for DPAA2, most MC commands have to be run through DPMCP's instead of the DPRC root container.

This requirement is enforced by QEMU which has a security filter: https://source.codeaurora.org/external/qoriq/qoriq-components/qemu/tree/hw/vfio/fsl_mc.c?h=integration&id=14fda5a42df6c72e890d6a97ff88c5852172604b#n688

If you attempt to do DPBP, DPIO or most other object instructions through the DPRC, you get a 0x3 (authentication error) response generated by QEMU:

dpaa2_mc0: mem 0x4040000000-0x404000ffff on ofwbus0
dpaa2_rc0: on dpaa2_mc0
dpaa2_rc0: MC firmware version: 10.20.4
dpaa2_bp0: at dpbp (id=0) on dpaa2_rc0
dpaa2_bp0: Failed to reset DPBP: id=0, error=3
device_attach: dpaa2_bp0 attach returned 6
dpaa2_io0: <DPAA2 I/O> iomem 0x4048000000-0x404800ffff,0x4044000000-0x404400ffff at dpio (id=0) on dpaa2_rc0
dpaa2_io0: Failed to reset DPIO: id=0, error=3
device_attach: dpaa2_io0 attach returned 6
dpaa2_con0: at dpcon (id=2) on dpaa2_rc0
dpaa2_con0: Failed to reset DPCON: id=2, error=3
device_attach: dpaa2_con0 attach returned 6 dpaa2_con1: at dpcon (id=0) on dpaa2_rc0
dpaa2_con1: Failed to reset DPCON: id=0, error=3

[dsalychev]

@mcbridematt Please, try 2cab65b0c135dd12cdf6172c11c7db8b227815e3 or any commit after that one. I've implemented a mechanism to discover and allocate DPMCPs.

[mcbridematt]

Thanks very much for that! Under VFIO it can finish boot now.

dpaa2_rc0: <DPAA2 Resource Container> on dpaa2_mc0
dpaa2_rc0: MC firmware version: 10.20.4
dpaa2_rc0: Resource container ID: 2
dpaa2_rc0: Objects in container: 8
dpaa2_rc0: Isolation context ID: 0
dpaa2_mcp0: <DPAA2 MC portal> iomem 0x4040020000-0x404002ffff at dpmcp (id=28) on dpaa2_rc0
dpaa2_mcp1: <DPAA2 MC portal> iomem 0x4040010000-0x404001ffff at dpmcp (id=27) on dpaa2_rc0
dpaa2_rc0: dpaa2_rc_discover: skip unsupported DPAA2 object: idx=2
dpaa2_bp0: <DPAA2 Buffer Pool> dpmcp (id=27) at dpbp (id=0) on dpaa2_rc0
dpaa2_io0: <DPAA2 I/O> iomem 0x4048000000-0x404800ffff,0x4044000000-0x404400ffff dpmcp (id=28) at dpio (id=0) on dpaa2_rc0
dpaa2_io0: using IRQ 40 for MSI
dpaa2_io0: dpio_id=0, swp_id=9, chan_mode=local_channel, notif_priors=2, swp_version=0x4010001
dpaa2_con0: <DPAA2 Concentrator> dpmcp (id=28) at dpcon (id=2) on dpaa2_rc0
dpaa2_con0: chan_id=3, priorities=2
dpaa2_con1: <DPAA2 Concentrator> dpmcp (id=27) at dpcon (id=0) on dpaa2_rc0
dpaa2_con1: chan_id=2, priorities=2
dpaa2_rc0: dpaa2_rc_discover: skip unsupported DPAA2 object: idx=2
dpaa2_ni0: <DPAA2 Network Interface> dpio (id=0) dpbp (id=0) dpcon (id=0) dpmcp (id=28) at dpni (id=1) on dpaa2_rc0
dpaa2_ni0: options=0x0 queues=1 tx_channels=0 wriop_version=0x422
dpaa2_ni0:      traffic classes: rx=1 tx=1 cgs_groups=1
dpaa2_ni0:      table entries: mac=16 vlan=0 qos=0 fs=64
dpaa2_ni0:      key sizes: qos=0 fs=56
dpaa2_ni0: Rx/Tx buffers: size=9216, alignment=64
dpaa2_ni0: Tx data offset=192
dpaa2_ni0: connected to dpmac (id=3)
dpaa2_ni0: dpaa2_ni_setup: failed to open connected DPMAC: 3 (assuming in other DPRC)
dpaa2_ni0: connected DPMAC is in FIXED mode
dpaa2_ni0: Ingress traffic classification is not supported
dpaa2_ni0: channels=1
dpaa2_ni0: channel: dpio_id=0 dpcon_id=0 chan_id=2, priorities=2
dpaa2_ni0: Ingress traffic distribution not supported
dpaa2_ni0: using IRQ 41 for MSI

dpni0 isn't receiving any frames (I can see frames coming out), but I will troubleshoot this another time.

It also appears we need to treat an unreachable (in other DPRC) DPNI partner as a 'fixed' link. I'll send a patch once I figure out why the ingress isn't working.

(edit: maybe due to an incorrect ingress filter?

dev.dpaa2_ni.0.stats.in_filtered_frames: 328
dev.dpaa2_ni.0.stats.in_discarded_frames: 0
dev.dpaa2_ni.0.stats.in_nobuf_discards: 145

I'll look at it again tomorrow )