marzoul
commented
6 years ago I don't understand the title of your which says bandwidth drops. With what and compared to what ? By how much ?
What I see is that you asked for 9999999999 words = 10 billion. Not only this overflows 32 bits (see the code of the app and try man atoi) but it also exceeds what Riffa is capable of per tranfer (2 billions, 31 bits). So getting this cropped somewhere (to 65536 it seems) is not a surprise ;-)
Just to be clear, in https://github.com/KastnerRG/riffa/pull/18 I didn't modify the way kernel paging mechanisms are used. I just applied the macros/functions renaming that the kernel developers did along kernel versions.
If you noticed a BW drop for all kernel versions, this makes me think of recent Spectre/Meltdown patches that suddenly got applied to ALL kernel versions. As this isolates kernel pages from other process pages (if you're using Intel CPU), the only impact this can have on perfs is a slowdown. Assuming this is the cause of course. In that case, I know there were discussions to add a kernel parameter so this this costly kernel page table isolation can be disabled, but I don't know the current status of this (if this parameter does exist, it's probably only for latest kernel anyway). You can always recompile the kernel by disabling this and give it a try.
Other than that, please check your system logs when an FPGA is discovered by the kernel. It prints the negociated speed and other interesting information. Your 1400/1500 MB/s seems like PCIe Gen1 2.5GT/s 8x to me, with BW significantly lower than theoretical 2GB/s. You should check with much larger buffers to mask sync latency and to get closer to what your machine can achieve (like 100MB at least).
buttercutter
drichmond
jixi2018
I have tried with linux kernel 4.4, 4.13 and 4.15.5 , testutil check is only giving me around 1.5Gbps for very large sample size.
@marzoul Does it have to do with the paging mechanism in the newer linux kernel as you had modified for #18 ? I suspect that it is something else because linux kernel 4.4 is also affected as per my experience. Could you advise ?
Is there a way to debug this bandwidth reduction issue using the linux driver compiled ("make debug") with debug support enabled ?
Why am I only getting 65536 words when I sent 99999 words ? kernel log for "./testutil 2 0 0 99999"