search

ipbus / ipbus-firmware

Firmware that implements a reliable high-performance control link for particle physics electronics, based on the IPbus protocol
https://ipbus.web.cern.ch
Other
41 stars 32 forks source link

Is there a standard/known benchmark for IPbus performance? #167

Closed jhegeman closed 4 years ago

jhegeman commented 4 years ago

Hi,

In our current design, based on a KU15P with IPbus over PCIe, we seem to observe unfeasibly slow single-register IPbus access. Is there a standard/known benchmark that can be used to verify that there is nothing wrong with our implementation?

Cheers, Jeroen

tswilliams commented 4 years ago

Hi

With the example designs I've previously measured latencies of around 40us for single-register access in a dev card housed in a standard computer in the lab.

In contrast on the Serenity with an Intel Atom-based COM express, the single-word latency is typically around 250us if the CPU is in 'performance' mode (which we typically use). If the CPU is in power-saving mode, instead the single-word latency is around 500us. Here, the CPU is set to non-power-saving mode as follows:

sudo cpupower frequency-set --governor performance

Does that help?

Cheers, Tom

tswilliams commented 4 years ago

BTW, forgot to mention: These latencies can be measured using the PerfTester.exe executable installed under /opt/cactus/bin/uhal/tests. The BandwidthRx and BandwidthTx modes of this executable repeatedly perform a fixed-size read/write of user-specified depth at a particular address and print out resulting performance numbers (depth, address and number of iterations specified through the -w, -b and -i options).

E.g. to measure the latency of a single-word read at base address 0x1000:

/opt/cactus/bin/uhal/tests/PerfTester.exe -t BandwidthRx -d  ipbuspcie-2.0:///dev/xdma0_h2c_0,/dev/xdma0_c2h_0 -b 0x1000 -w 1 -i 1000
jhegeman commented 4 years ago

Hi Tom,

Thanks for the pointers. While I cannot reproduce your numbers exactly (which is not that strange since I don't have the exact same setup available), I confirm the relative estimates. Especially 'true' single register access (i.e., with a dispatch after every access) stands out in terms of (lack of) speed. This just means that if one has to poll something, the access becomes the limiting factor.

In any case it looks as if there is nothing fundamental wrong with my IPbus implementation/connection then.

Cheers, Jeroen