search

openucx / ucx

Unified Communication X (mailing list - https://elist.ornl.gov/mailman/listinfo/ucx-group)
http://www.openucx.org
Other
1.17k stars 428 forks source link

Shared memory performance in UCX appears worse than vader btl #2784

Open paklui opened 6 years ago

paklui commented 6 years ago

I compare the performance on vader BTL using XPMEM and KNEM, it shows that without UCX, vader BTL appears to show better shared memory performance than with UCX.

XPMEM's 4KB to 512KB is better than KNEM. Form my observation is UCX support for XPMEM and KNEM is always worse than vader BTL. (Note: I am NOT using HPC-X build, but building XPMEM/KNEM support directly from UCX)

Please look at the results for the comparisons

Another request is, for vader BTL, adjusting to 64KB or 256KB appear to be better. There is an option to adjust message sizes for the threshold for vader BTL but not UCX. As a suggestion for future release, is there a way to enable such option? For instance, in some platform by changing the KNEM vader BTL eager limit to 64KB and 256KB and appeared to show better bandwidth. Could we set the same option for UCX when KNEM is used? looking for the equivalent in UCX: -mca btl_vader_single_copy_mechanism knem -mca btl_vader_eager_limit 65536 ompi-shared-memory-ucx-vs-vader.xlsx

yosefe commented 6 years ago

UCX osu_bw result on shared memory/ARM is worse than OMPI/vader

shamisp commented 6 years ago

what is this ?

On Mon, Aug 20, 2018 at 6:35 AM Yossi Itigin notifications@github.com wrote:

https://camo.githubusercontent.com/193766a3b9959c5f4ed5cd8cf3251d015e839d51/68747470733a2f2f6769746875622e7472656c6c6f2e73657276696365732f696d616765732f6d696e692d7472656c6c6f2d69636f6e2e706e67 UCX osu_bw result on shared memory/ARM is worse than OMPI/vader https://trello.com/c/v9f8QpUG/10-ucx-osubw-result-on-shared-memory-arm-is-worse-than-ompi-vader

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openucx/ucx/issues/2784#issuecomment-414285821, or mute the thread https://github.com/notifications/unsubscribe-auth/ACIe2Oj9rp4uxRjkyy9LO421AgNgfjPDks5uSp8MgaJpZM4Vs0Oj .