search

xrootd / xrootd

The XRootD central repository https://my.cdash.org/index.php?project=XRootD
http://xrootd.org
Other
147 stars 149 forks source link

Performance tuning plots #2281

Open ivukotic opened 3 weeks ago

ivukotic commented 3 weeks ago

As somebody who has to tune xcaches for optimal performance, I would very much appreciate if every xrootd release would come with a set of curves showing throughput rate vs block size for both xrdcp and "analysis like" accesses.

amadio commented 3 weeks ago

Could you please elaborate on what you mean? Have you tried using the recorder plugin for the client? I am thinking about creating some way to easily plot the output coming from that plugin, but I'm not sure this is what you need. Cheers,

rob-c commented 3 weeks ago

Hi, We're building an XRootD testing setup at Edinburgh and may be able to contribute to this, if there's interest?

We're primarily working on setting up a multinode test setup for functionality testing to support UK sites. Ideally we would like to provide a platform to reproduce any problems reported by a UK XRootD site.

(Using k8s as a deployment platform partly as an excuse to build some familiarity within our site for more advanced container orchestration strategies/management).

I've been eyeing up using this to build a reproducible CI giving us "average performance vs X" between builds, if there's interest in this please let us know.

ivukotic commented 3 weeks ago

I have never heard about the recorder thing. Can you please point me to some documentation?

What I would really want is that each time there is a new build, a new series of xcache performance tests is performed. These tests should vary the way data is read (xrdcp, 90%, 50%, 30%, 10%), block size, latency to origin, just to mention a few. Plots of these results should be provided on the xrootd page. Also evolution of these values over time.

amadio commented 3 weeks ago

This is an interesting idea to be discussed in the upcoming XRootD and FTS Workshop. I would be very happy to have such tests as well. There are some tests ongoing with EOS in the context of RNTuple as well, it's a collaboration between CERN IT and EP-SFT. Some of these tests will involve xcache.