Open UCX

[shamisp]

---

Software Name

Open UCX

---

Public URL

https://github.com/openucx/ucx

---

Technical Overview

Summary: UCX is a communication library implementing high-performance messaging Description: UCX stands for Unified Communication X. It requires either RDMA-capable device (InfiniBand, RoCE, etc), Cray Gemini or Aries, for inter-node communication. Future versions will support also TCP for inter-node, to lift that hardware dependency. In addition, the library can be used for intra-node communication by leveraging the following shared memory mechanisms: posix. sysv, cma, knem, xpmem.

---

Latest stable version number

1.2.1 https://github.com/openucx/ucx/releases/tag/v1.2.1

---

Open-source license type

BSD-3

---

Relationship to component?

• [X] contributing developer
• [ ] user
• [ ] other

If other, please describe:

---

Build system

• [X] autotools-based
• [ ] CMake
• [ ] other

If other, please describe:

Does the current build system support staged path installations? For example: make install DESTIR=/tmp/foo (or equivalent)

• [X ] yes
• [ ] no

---

Does component run in user space or are administrative credentials required?

• [X] user space
• [ ] admin

---

Does component require post-installation configuration.

• [ ] yes
• [X] no

If yes, please describe briefly:

---

If component is selected, are you willing and able to collaborate with OpenHPC maintainers during the integration process?

• [X] yes
• [ ] no

---

Does the component include test collateral (e.g. regression/verification tests) in the publicly shipped source?

• [X] yes
• [ ] no

If yes, please briefly describe the intent and location of the tests.

---

Does the component have additional software dependencies (beyond compilers/MPI) that are not part of standard Linux distributions?

• [ ] yes
• [X] no

If yes, please list the dependencies and associated licenses.

---

Does the component include online or installable documentation?

• [X] yes
• [ ] no

If available online, please provide URL.

---

[Optional]: Would you like to receive additional review feedback by email?

- [X] yes - [ ] no

[rengolin]

Hi Pasha, thanks for the submission!

IIUC, UCX is a layer below MPI/SHMEM that allows you to better use the busses available, which is a good addition to our stack, providing it works well with our supported OpenMPI, MPICH and MVAPICH offerings.

A few questions below.

1. Is there any compatibility issues with other (competing) offerings that would require us to provide a choice to the users? For instance, we offer users the choice of MPI library (via module load compiler-mpi). In this case, would we need to extend the solution space to compiler-mpi-ucx?

2. If we install UCX as a dependency to any existing MPI offering, would it allow users to use/not use depending on the workload? Ie. are there any flags that need to be used, or is the discovery of what works with what automatic and transparent?

These two questions are regarding how we organise the dependencies, development environment and user documentation, and are important in the case we create a default behaviour that is different to the current one.

3. Does it need special hardware? If so, then we'll have to make sure we have those in the lab to test, or we can't ship it. Currently, the hardware we have available in the lab is heterogeneous, so x86 and Arm packages are different.

4. How would this integrate with our existing test-suite? Right now we install the environment, pick a toolchain-mpi and run the tests on the compute nodes (including MPI). If the installation/setup is not transparent, we'll have to add those to the recipes, CI scripts and make sure we test with and without it (basically doubling the tests), which is not trivial.

Finally this has to do with hardware availability and timing of release validation, which we're already stretching. If we can fit this in without new behaviour, we may not need to increase the CI load. But if not, we'll probably have to add more hardware before we can start testing this package (and shipping it).

[tgamblin]

@shamisp: did you see @rengolin's questions?

[shamisp]

@tgamblin yes, I'm a bit behind on my GitHub responses. I will try to follow this week.

[shamisp]

1. I don't think there is a dependency like compiler-ucx, but I think there is a compile (?) time dependency mpi-ucx for MPICH which can be build with portals, and intel ofi as well. Adding @raffenet to comment on this. For OpenMPI there is no such dependency issue since it can be build with multiple communication library options and the selection happens during the runtime (based on internal OpenMPI priorities).

2. For OpenMPI the selection is transparent , but there is also runtime option.

3. Tested (CC) on Power, Arm-v8, x86, GPUs (Nvidia, AMD). UCX does not require special hardware for testing. We auto-detect underlaying hardware and use whatever is available. Even if networking hardware is not available it can use local memory (posix, sysv, cma, kneel, xpmem, cuda, Rocm) for communication. In terms of inter-node communication we support RoCE, InfiniBand, uGNI and limited support for TCP (not all features are available)

4. You can run full CC testing with make -C test/gtest test , no special magic required. Based on the underlaying hardware it will run between 1k-4k tests. To accelerate execution of the testing you can use some of the parallel run options that we have. More examples here: https://github.com/openucx/ucx/blob/master/contrib/test_jenkins.sh

[shamisp]

@yosefe - FYI.

[raffenet]

UCX support for MPICH must be selected at configure time.

--with-device=ch4:ucx --with-ucx=<path/to/install>

Note that this configuration will be supported in MPICH >= 3.3, which has not done a full release yet. There are alphas currently available, though.

[shamisp]

@raffenet Then I would say that we are good for now. Once MPICH is bumped up to 3.3, OpenHPC anyways would have deal with the issues regardless of OpenUCX.

[koomie]

Thank you for the submission and extra details on the follow up questions. The TSC review panel was generally favorable to the inclusion, but recommends to put on hold until the underlying companion dependency for MPICH is released. As was pointed out, once that full release is out, we will have to deal with the move to ch4 anyway; tagging this as a 'Reject Until Changes' for now, but the change is simply tied to the formal MPICH release. Once that is out and we start to include for future ohpc release, the plan is to then pop UCX onto the backlog.

[alex--m]

@shamisp Seems CH4 is part of the mpich-3.3 stable release (last November). Does this mean UCX submission could now be reconsidered?

[koomie]

Yes, I will add the latest MPICH to the backlog which should allow builds for UCX as well. Quick question - can this latest MPICH support monolithic builds that support CH4 and previous transports (can we have one build that works on tcp and IB)?

[raffenet]

@koomie we do support some monolithic builds.

With the ch3 device, we support compiling in support for multiple networks at once:

--with-device=ch3:nemesis:tcp,mxm

for example will default to the TCP for internode communication. MXM (Infiniband) can be used if MPIR_CVAR_NEMESIS_NETMOD=mxm is set in the running environment.

With ch4, we currently only support ofi and ucx netmods. You can build both at the same time

--with-device=ch4:ofi,ucx

but it is recommended to build a single netmod for performance reasons. TCP is supported in ch4 indirectly by either libfabric or ucx tcp support. Simultaneous builds of ch3 and ch4 are not possible.

Hope that helps. Happy to take additional questions.

[koomie]

Thanks @raffenet; given that newness of ch4, would you possibly be interested/willing to give our technical steering committee a short overview (would guess order 10-15 minutes)? I suspect we may have more questions as we work to roll up a ch4 build. Our next meeting is this Wednesday (10-11 central) and I can work around your schedule if you'd prefer to go first or later in the session.

[raffenet]

Sure, I am available to give an overview this Wednesday.

[koomie]

Awesome - I just sent meeting details separately thru email.