bosilca
commented
4 years ago Yea, both these features are fully supported. Let me first talk about the second one, the configuration file passed through the coll_tuned_dynamic_rules_filename MCA parameter. The format is described in the paper you pointed, but there are many examples in our mailing list. It supports all collectives provided by tuned, for as long as tuned is the module selected for a particular collective (you can enforce this by setting tuned priority to 100). For the sake of completeness here is another example that fiddle with the MPI_Alltoall and MPI_Allreduce collectives:
2 # num of collectives
3 # ID = 3 Alltoall collective (ID in coll_tuned.h)
# 0: ignore, 1: basic linear, 2: pairwise, 3: modified bruck,
# 4: linear with sync, 5:two proc only
1 # number of com sizes
64 # comm size 64
2 # number of msg sizes
1024 3 0 0 # for message size 0, bruck 1, topo 0, 0 segmentation
8192 2 0 0 # 8k+, pairwise 2, no topo or segmentation
# end of first collective
2 # ID = 2 Allreduce collective (ID in coll_tuned.h)
1 # number of com sizes
8 # comm sizes 8
2 # number of msg sizes
0 1 0 0 # for message size 0, basic linear 1, topo 0, 0 segmentation
1024 2 0 0 # for messages size > 1024, nonoverlapping 2, topo 0, 0 segmentation
# end of collective ruleThe first approach, via coll_tuned_<COLL NAME>_algorithm might have less flexibility to precisely select collective for ranges of processes or message size, but can be dynamically changed during the execution by setting the corresponding MCA parameter right before creating a new communicator. As a result you can finely tailor the collective algorithm behind each collective on the communicators that matter most for your application.
miharulidze
mkurnosov
yqin
wckzhang
Background information
What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)
v4.0.3
Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
Distribution tarball from open-mpi.org.
./configure --prefix=$(pwd)/build --with-ucx=/path-to-ucx-installation/ --enable-orterun-prefix-by-defaultPlease describe the system on which you are running
Details of the problem
Dear OpenMPI developers,
I'm trying to provide tuned selection of collective algorithms for
tunedcomponent. It seems like there are two ways: 1) through command line options like--mca coll_tuned_use_dynamic_rules 1 --mca coll_tuned_<COLL NAME>_algorithm <ALGORITHM ID>. This method works fine and I notice the big difference between algorithms while running OSU benchmarks. At the same time, this method not allows me to do fine-grained tuning, like specifying a communicator size, message thresholds, etc. 2) Specify algorithm selection policy using file with the following options--mca coll_tuned_use_dynamic_rules 1 --mca coll_tuned_dynamic_rules_filename <PATH TO ALGORITHM RULES>. This paper (actually, this the only example of rules file I found in Google) shows an example of Alltoall algorithm selection tuning for different use cases. I also done some experiments with Alltoall, but it showed no difference between several algorithms/thresholds at all.Here is my questions: 1) Is this feature still supported in tuned component? 2) Are there some restrictions to use it, for example it works only with subset of collectives that are implemented in tuned component? 3) Are there any examples of rules file for other operations?
I'll be grateful for any help. Thank you in advance!