search

Microsemi / switchtec-kernel

A kernel module for the Microsemi PCIe switch
GNU General Public License v2.0
45 stars 31 forks source link

ntb_perf with dma flag issue #123

Closed fatdenny closed 3 months ago

fatdenny commented 3 months ago

Hi all,

I am working with "PFX 68xG4" with ntb module. ntb_perf without dma, it works.

kernel: 5.15.0-107-generic Here is the log :

Device# modprobe ntb_perf max_mw_size=0 chunk_order=19 total_order=30 use_dma=0 Host # modprobe ntb_perf max_mw_size=0 chunk_order=19 total_order=30 use_dma=0

Device# echo 0 > /sys/kernel/debug/ntb_perf/*/run
Device# cat /sys/kernel/debug/ntb_perf/*/run
    Peer 0 test statistics:
    0: copied 1073741824 bytes in 101326 usecs, 10596 MBytes/s

Host  # echo 0 > /sys/kernel/debug/ntb_perf/*/run
Host  # cat /sys/kernel/debug/ntb_perf/*/run
    Peer 0 test statistics:
    0: copied 1073741824 bytes in 142943 usecs, 7511 MBytes/s

But , it doesn't work with dma flag

Device# modprobe ntb_perf max_mw_size=0 chunk_order=19 total_order=30 use_dma=1
Host  # modprobe ntb_perf max_mw_size=0 chunk_order=19 total_order=30 use_dma=1   

Device# echo 4 > /sys/kernel/debug/ntb_perf/*/threads_count
Device# echo 0 > /sys/kernel/debug/ntb_perf/*/run
Device# cat /sys/kernel/debug/ntb_perf/*/run

root@pegasus-desktop:~/switchtec-kernel# cat /sys/kernel/debug/ntb_perf/*/run
Peer 0 test statistics:
0: error status -19
1: error status -19
2: error status -19
3: error status -19

How can I use dma in ntb_perf module ? Thank you.

BR Denny

lsgunth commented 3 months ago

Does your system have a DMA engine to use? Switchtec hardware doesn't have one. So in order to use dma you need a system with Intel's IOAT or similar.

fatdenny commented 3 months ago

Thank Logan's info.

After load dma module, ntb_perf with dma flag works.

Device# modprobe ntb_perf max_mw_size=0 chunk_order=19 total_order=30 use_dma=1 Device# echo 0 > /sys/kernel/debug/ntb_perf//run Device# cat /sys/kernel/debug/ntb_perf//run Peer 0 test statistics: 0: copied 1073741824 bytes in 79438 usecs, 13516 MBytes/s

dmesg: kernel: [ 1507.092789] switchtec_dma:switchtec_dma_alloc_chan_resources:1334: dma dma0chan0: Burst Size: 0x6 kernel: [ 1507.111906] switchtec_dma:switchtec_dma_alloc_chan_resources:1337: dma dma0chan0: Burst Scale: 0x1 kernel: [ 1507.131019] switchtec_dma:switchtec_dma_alloc_chan_resources:1340: dma dma0chan0: Interval: 0x0 kernel: [ 1507.150155] switchtec_dma:switchtec_dma_alloc_chan_resources:1343: dma dma0chan0: Arb Weight: 0x1 kernel: [ 1507.169278] switchtec_dma:switchtec_dma_alloc_chan_resources:1346: dma dma0chan0: MRRS: 0x3