Open krakowski opened 3 years ago
@krakowski is it possible there are many unexpected tagged messages which are not being received? it's possible to heck this by probing with tag_mask=0 : call it repeatedly drain the unexpected queue.
Also, it is possible some send/recv requests are not being released? Is there any warning about leaked objects during ucp_worker_destroy()?
Hi @yosefe,
Running the benchmark for a short amount of time works and no warnings about leaked resources are logged. Before running the benchmark I exchange some messages for setting up the environment and synchronization, but everything runs in a single thread, so these calls should not be reached a second time once setup (connection establishment and exchange of remote keys) is finished.
Does ucp_atomic_op_nbx
also send tagged messages I have to poll or are there any other resources (except for the request, if it is one) I have to release after this function call?
ucp_atomic_op_nbx does not send a tagged message.. can you double check that request_free is actually called the expected # of times? another way - build ucx with debug mode (./contrib/configure-devel) and upload log file with UCX_LOG_LEVEL=data. Then we can see the lifetime of all objects and why they are not destroyed.
Just checked this. The pointers I get from ucp_atomic_op_nbx
are all UCS_OK
(null pointers). From what I read this means that the request finished immediately. Do I have to call ucp_request_free
on them, too? At the moment I check if the returned value/pointer is within the range of a status and only call ucp_request_free
on it if it isn't.
Just tried it. Calling ucp_request_free
on them results in a segfault.
I think the issue is that too many requests are being queued internally. Will try to provide a fix for it which will create some backpressure (so at some point, will return a request instead of UCS_OK)
I can check this by waiting some amount of time between calls to ucp_atomic_op_nbx
and see if the memory consumption reduces. Will report my results soon.
Do you still need the log output with debug mode? I could try to build the RPM packages in debug mode and run my benchmark with UCX_LOG_LEVEL=data
.
I can check this by waiting some amount of time between calls to
ucp_atomic_op_nbx
and see if the memory consumption reduces. Will report my results soon.Do you still need the log output with debug mode? I could try to build the RPM packages in debug mode and run my benchmark with
UCX_LOG_LEVEL=data
.
No need for now. As another experiment, i'd suggest to try a blocking {ucp_ep_flush_nbx+poll_wait} every 1k iterations or so.
@yosefe Your suggestion is working! :)
Calling ucp_ep_flush_nbx
and waiting for it to complete after every 1000 invocations of ucp_atomic_op_nbx
keeps the memory consumption steady. I also got better performance results (~0.95 us
per atomic add) presumably by preventing new memory allocations.
Since I am already talking to someone knowing how ucx works I would like to ask another off-topic question, if this is ok. Is it normal that RDMA reads are a lot slower than RDMA writes? With a buffer size of 64
bytes I get ~0.2us
for writes while reads take ~2.0us
(until ucp_request_check_status
reports UCS_OK
).
Writes completed locally the moment the user data is copied to the device or local bouncing buffer. For read to be completed (aka data delivered to the user) you have to wait for a full roundtrip.
@shamisp Thanks for the explanation! :)
I already thought something along those lines. Is it then somehow possible to measure write latency at the initiators side without adding extra overhead? Since the receiver is not aware of the RDMA write, I can only think of polling the memory region for changes and sending a message back if a change is detected. Unfortunately, this would add some considerable amount of overhead to the measurement.
@krakowski We already have such test implemented (for both uct/ucp layers) in ucx_perftest. UCT/UCP put latency test measures half round trip latency using ping-pong protocol, which is similar to what you describe.
@shamisp Thanks, I will take a look at it and try to port it. I should have mentioned that I am using a custom Java binding for UCX which I want to benchmark. This is why I search for a way to reliably measure the latency :slightly_smiling_face:
Adding @petro-rudenko who implemented the Java binding. Maybe he has some benchmarks to share.
Describe the bug
I use Oracle's Project Panama (which will replace JNI in the near feature) to generate Java bindings for ucp. Right now I am in the process of benchmarking my implementation using JMH. Sending messages using
ucp_tag_send_nbx
anducp_stream_send_nbx
as well as accessing remote memory usingucp_put_nbx
anducp_get_nbx
is working and gives decent results. Continuously callingucp_atomic_op_nbx
on the other hand fills the local RAM in a matter of seconds and therefore crashes the process.I used jemalloc to see if I can identify which function is responsible for the huge amounts of allocated memory and found that
ucs_mpool_hugetlb_malloc
could be it.Here is the top of the report jemalloc generated (before the process was killed).
If you are interested in the Java code, here are the relevant parts called during the Benchmark.
LatencyBenchmark#atomicAdd
AtomicIntegerContext#blockingAtomicAdd
BenchmarkConnection#blockingAtomicAdd32
Endpoint#atomic
All parameters passed to
ucp_tag_send_nbx
are allocated exactly once during the benchmark.Steps to Reproduce
Perform the following steps in a loop
ucp_atomic_op_nbx
withUCP_ATOMIC_OP_ADD
ucp_request_check_status
ucp_request_free
Operating System
CentOS Linux release 8.1.1911 (Core)
4.18.0-277.el8.x86_64
Packages
rdma-core-32.0-4.el8.x86_64
ucx-1.9.0-1.el8.x86_64
ucx-ib-1.9.0-1.el8.x86_64
ucx-rdmacm-1.9.0-1.el8.x86_64
ucx-cma-1.9.0-1.el8.x86_64
ucx-devel-1.9.0-1.el8.x86_64
Hardware
ibstat
Additional information
ucx_info -d
ucx_info -b