search

NVIDIA / CUDALibrarySamples

CUDA Library Samples
Other
1.5k stars 311 forks source link

Runtime Error in cuSOLVERMp (RDMA_CM_EVENT_ROUTE_ERROR) #190

Closed s769 closed 2 months ago

s769 commented 2 months ago

I am trying to set up cuSOLVERMp on my HPC system (Lonestar 6 at TACC). I loaded the HPC SDK module

Currently Loaded Modules: 
1) intel/19.1.1   2) hpcx   3) nvhpc-hpcx-cuda12/24.5

I successfully compiled the cuSOLVERMp examples with make

I then ran one of the examples and got the following errors. The other examples show similar behavior.

c306-001.ls6(1135)$ mpiexec -np 2 mp_potrf_potrs -verbose 1
Parameters: m=1 n=10 nrhs=1 mbA=2 nbA=2 mbB=2 nbB=2 mbQ=2 nbQ=2 mbZ=0 nbZ=0ia=3 ja=3 ib=3 jb=1 iq=1 jq=1 iz=0 jz=0 p=2 q=1 grid_layout= verbose=1
Parameters: m=1 n=10 nrhs=1 mbA=2 nbA=2 mbB=2 nbB=2 mbQ=2 nbQ=2 mbZ=0 nbZ=0ia=3 ja=3 ib=3 jb=1 iq=1 jq=1 iz=0 jz=0 p=2 q=1 grid_layout= verbose=1
[1718821929.555491] [c306-001:2563497:0]          ucc_mc.c:78   UCC  WARN  mc cpu mc was allready initilized with different thread mode: current tm 0, provided tm 2
[1718821929.555524] [c306-001:2563497:0]          ucc_mc.c:78   UCC  WARN  mc cuda mc was allready initilized with different thread mode: current tm 0, provided tm 2
[1718821929.555529] [c306-001:2563497:0]          ucc_ec.c:75   UCC  WARN  ec cpu ec was allready initilized with different thread mode: current tm 0, provided tm 2
[1718821929.555535] [c306-001:2563497:0]          ucc_ec.c:75   UCC  WARN  ec cuda ec was allready initilized with different thread mode: current tm 0, provided tm 2
[1718821929.555491] [c306-001:2563496:0]          ucc_mc.c:78   UCC  WARN  mc cpu mc was allready initilized with different thread mode: current tm 0, provided tm 2
[1718821929.555525] [c306-001:2563496:0]          ucc_mc.c:78   UCC  WARN  mc cuda mc was allready initilized with different thread mode: current tm 0, provided tm 2
[1718821929.555530] [c306-001:2563496:0]          ucc_ec.c:75   UCC  WARN  ec cpu ec was allready initilized with different thread mode: current tm 0, provided tm 2
[1718821929.555537] [c306-001:2563496:0]          ucc_ec.c:75   UCC  WARN  ec cuda ec was allready initilized with different thread mode: current tm 0, provided tm 2
[c306-001][Jun 19 13:32:10 943233][RDMA_SR][2563496][error] - Error event recieved: event: RDMA_CM_EVENT_ROUTE_ERROR,  error: -22
[c306-001][Jun 19 13:32:10 943324][RDMA_SR][2563496][error] - Error occured during connection event handle
[c306-001][Jun 19 13:32:13 946373][RDMA_SR][2563496][error] - poll failed due to poll_timeout=3000.000000, stop
[c306-001][Jun 19 13:32:13 946440][RDMA_SR][2563496][error] - Poll failed
[c306-001][Jun 19 13:32:13 946448][RDMA_SR][2563496][error] - Failed to connect
[c306-001][Jun 19 13:32:13 946551][RDMA_SR][2563496][error] - rdma_resolve_addr failed with error: -1
[c306-001][Jun 19 13:32:13 946568][RDMA_SR][2563496][error] - rdma_resolve_addr failed with error: -1
[c306-001][Jun 19 13:32:13 946581][GENERAL][2563496][error] - unable to query AM service record(AM query)
[c306-001][Jun 19 13:32:15 185169][RDMA_SR][2563496][error] - Error event recieved: event: RDMA_CM_EVENT_ROUTE_ERROR,  error: -22
[c306-001][Jun 19 13:32:15 185208][RDMA_SR][2563496][error] - Error occured during connection event handle
[c306-001][Jun 19 13:32:18 188262][RDMA_SR][2563496][error] - poll failed due to poll_timeout=3000.000000, stop
[c306-001][Jun 19 13:32:18 188326][RDMA_SR][2563496][error] - Poll failed
[c306-001][Jun 19 13:32:18 188334][RDMA_SR][2563496][error] - Failed to connect
[c306-001][Jun 19 13:32:18 188404][RDMA_SR][2563496][error] - rdma_resolve_addr failed with error: -1
[c306-001][Jun 19 13:32:18 188421][RDMA_SR][2563496][error] - rdma_resolve_addr failed with error: -1
[c306-001][Jun 19 13:32:18 188430][GENERAL][2563496][error] - unable to query AM service record(AM query)
print_host_matrix : Input matrix A
-nan  -nan  -nan  -nan  -nan  -nan  -nan  -nan  -nan  -nan
-nan  -nan  -nan  -nan  -nan  -nan  -nan  -nan  -nan  -nan
11.72  0.39  0.78  0.80  0.91  0.20  0.34  0.77  0.28  0.55
0.39  10.69  0.63  0.36  0.51  0.95  0.92  0.64  0.72  0.14
0.78  0.63  4.99  0.02  0.24  0.14  0.80  0.16  0.40  0.13
0.80  0.36  0.02  7.17  1.00  0.22  0.51  0.84  0.61  0.30
0.91  0.51  0.24  1.00  7.38  0.52  0.49  0.97  0.29  0.77
0.20  0.95  0.14  0.22  0.52  5.74  0.77  0.40  0.89  0.28
0.34  0.92  0.80  0.51  0.49  0.77  4.30  0.81  0.92  0.07
0.77  0.64  0.16  0.84  0.97  0.40  0.81  3.12  0.53  0.09
0.28  0.72  0.40  0.61  0.29  0.89  0.92  0.53  1.71  0.66
0.55  0.14  0.13  0.30  0.77  0.28  0.07  0.09  0.66  1.78
print_host_matrix : Input matrix X
-nan
-nan
-nan
-nan
-nan
-nan
-nan
-nan
-nan
-nan
-nan
-nan
print_host_matrix : Input matrix B
-nan
-nan
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
print_host_matrix : Output matrix X
-nan
-nan
0.03
0.04
0.14
0.07
0.03
0.08
0.09
0.20
0.19
0.42

|b - A*x|_inf = 2.220446E-16
|x|_inf = 1.273718E+00
|b|_inf = 1.000000E+01
|A|_inf = 1.674133E+01
|b - A*x|/(|A|*|x|+|b|) = 7.088702E-18

Aside from the error in the middle, it looks like the program ran successfully, but the verbose output shows some NaNs were present in the computations.

If I try running with -np 3, the program hangs after the [c306-001][Jun 19 13:32:18 188430][GENERAL][2563496][error] - unable to query AM service record(AM query) line.

I would appreciate any help to resolve this.

mrogowski commented 2 months ago

Note ia=3 ja=3 ib=3 jb=1 you have in your parameters. Those specify the first row/column of A/B that will be used (see the documentation). That's why you get nan - the memory in the first two rows was never touched. You will not see those two rows if you specify ia=1 ja=1 ib=1 jb=1.

mrogowski commented 2 months ago

As for the hang with 3 processes, please see this comment - p*q has to be equal to the number of processes you are using (and equal to the number of GPUs). Are you running -np 3 with p=2 q=1?

s769 commented 2 months ago

Thank you. I changed the parameters you mentioned and now the example with 2 GPUs also works

c318-004.ls6(1011)$ mpiexec -np 2 mp_potrf_potrs -verbose 1 -ia 1 -ib 1 -ja 1 -jb 1
Parameters: m=1 n=10 nrhs=1 mbA=2 nbA=2 mbB=2 nbB=2 mbQ=2 nbQ=2 mbZ=0 nbZ=0ia=1 ja=1 ib=1 jb=1 iq=1 jq=1 iz=0 jz=0 p=2 q=1 grid_layout= verbose=1
Parameters: m=1 n=10 nrhs=1 mbA=2 nbA=2 mbB=2 nbB=2 mbQ=2 nbQ=2 mbZ=0 nbZ=0ia=1 ja=1 ib=1 jb=1 iq=1 jq=1 iz=0 jz=0 p=2 q=1 grid_layout= verbose=1
[1719257178.325559] [c318-004:275852:0]          ucc_mc.c:78   UCC  WARN  mc cpu mc was allready initilized with different thread mode: current tm 0, provided tm 2
[1719257178.325572] [c318-004:275852:0]          ucc_mc.c:78   UCC  WARN  mc cuda mc was allready initilized with different thread mode: current tm 0, provided tm 2
[1719257178.325576] [c318-004:275852:0]          ucc_ec.c:75   UCC  WARN  ec cpu ec was allready initilized with different thread mode: current tm 0, provided tm 2
[1719257178.325580] [c318-004:275852:0]          ucc_ec.c:75   UCC  WARN  ec cuda ec was allready initilized with different thread mode: current tm 0, provided tm 2
[1719257178.325559] [c318-004:275853:0]          ucc_mc.c:78   UCC  WARN  mc cpu mc was allready initilized with different thread mode: current tm 0, provided tm 2
[1719257178.325573] [c318-004:275853:0]          ucc_mc.c:78   UCC  WARN  mc cuda mc was allready initilized with different thread mode: current tm 0, provided tm 2
[1719257178.325576] [c318-004:275853:0]          ucc_ec.c:75   UCC  WARN  ec cpu ec was allready initilized with different thread mode: current tm 0, provided tm 2
[1719257178.325581] [c318-004:275853:0]          ucc_ec.c:75   UCC  WARN  ec cuda ec was allready initilized with different thread mode: current tm 0, provided tm 2
[c318-004][Jun 24 14:26:19 503804][RDMA_SR][275852][error] - Error event recieved: event: RDMA_CM_EVENT_ROUTE_ERROR,  error: -22
[c318-004][Jun 24 14:26:19 503874][RDMA_SR][275852][error] - Error occured during connection event handle
[c318-004][Jun 24 14:26:22 506892][RDMA_SR][275852][error] - poll failed due to poll_timeout=3000.000000, stop
[c318-004][Jun 24 14:26:22 506922][RDMA_SR][275852][error] - Poll failed
[c318-004][Jun 24 14:26:22 506928][RDMA_SR][275852][error] - Failed to connect
[c318-004][Jun 24 14:26:22 506983][RDMA_SR][275852][error] - rdma_resolve_addr failed with error: -1
[c318-004][Jun 24 14:26:22 506994][RDMA_SR][275852][error] - rdma_resolve_addr failed with error: -1
[c318-004][Jun 24 14:26:22 507005][GENERAL][275852][error] - unable to query AM service record(AM query)
[c318-004][Jun 24 14:26:23 738810][RDMA_SR][275852][error] - Error event recieved: event: RDMA_CM_EVENT_ROUTE_ERROR,  error: -22
[c318-004][Jun 24 14:26:23 738838][RDMA_SR][275852][error] - Error occured during connection event handle
[c318-004][Jun 24 14:26:26 741854][RDMA_SR][275852][error] - poll failed due to poll_timeout=3000.000000, stop
[c318-004][Jun 24 14:26:26 741883][RDMA_SR][275852][error] - Poll failed
[c318-004][Jun 24 14:26:26 741890][RDMA_SR][275852][error] - Failed to connect
[c318-004][Jun 24 14:26:26 741939][RDMA_SR][275852][error] - rdma_resolve_addr failed with error: -1
[c318-004][Jun 24 14:26:26 741950][RDMA_SR][275852][error] - rdma_resolve_addr failed with error: -1
[c318-004][Jun 24 14:26:26 741956][GENERAL][275852][error] - unable to query AM service record(AM query)
print_host_matrix : Input matrix A
11.72  0.39  0.78  0.80  0.91  0.20  0.34  0.77  0.28  0.55
0.39  10.69  0.63  0.36  0.51  0.95  0.92  0.64  0.72  0.14
0.78  0.63  4.99  0.02  0.24  0.14  0.80  0.16  0.40  0.13
0.80  0.36  0.02  7.17  1.00  0.22  0.51  0.84  0.61  0.30
0.91  0.51  0.24  1.00  7.38  0.52  0.49  0.97  0.29  0.77
0.20  0.95  0.14  0.22  0.52  5.74  0.77  0.40  0.89  0.28
0.34  0.92  0.80  0.51  0.49  0.77  4.30  0.81  0.92  0.07
0.77  0.64  0.16  0.84  0.97  0.40  0.81  3.12  0.53  0.09
0.28  0.72  0.40  0.61  0.29  0.89  0.92  0.53  1.71  0.66
0.55  0.14  0.13  0.30  0.77  0.28  0.07  0.09  0.66  1.78
print_host_matrix : Input matrix X
-nan
-nan
-nan
-nan
-nan
-nan
-nan
-nan
-nan
-nan
print_host_matrix : Input matrix B
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
print_host_matrix : Output matrix X
0.03
0.04
0.14
0.07
0.03
0.08
0.09
0.20
0.19
0.42

|b - A*x|_inf = 2.220446E-16
|x|_inf = 1.273718E+00
|b|_inf = 1.000000E+01
|A|_inf = 1.674133E+01
|b - A*x|/(|A|*|x|+|b|) = 7.088702E-18

Setting p and q correctly should also work (I was using incompatible values before). I still see the error messages in the middle though (even if the output is correct). Do you know how to address those?

mrogowski commented 2 months ago

I changed the parameters you mentioned and now the example with 2 GPUs also works

To clarify, it worked before. 🙂 It's a feature, not a bug - ia, ja allows you to perform operations on a submatrix if needed. ScaLAPACK explains it well: https://www.netlib.org/scalapack/slug/node79.html.

As for the RDMA error, I am not sure. Probably UCC tries to initialize a backend that doesn't work on your system. UCC tries to initialize what it can, and if it fails it falls back to a working backend. This would explain why the application works correctly.

My suggestions that may help narrow it down are:

  1. Run after setting export UCC_TLS=^mlx5,sharp to check if it's coming from one of those layers.
  2. export UCC_LOG_LEVEL=DEBUG for extra UCC debugging output to see if that gives us any clues.
  3. Ask your system administrator if they have seen this before.
s769 commented 2 months ago

Here is the output with those environment variables set.

c318-001.ls6(1007)$ mpiexec -np 2 mp_potrf_potrs -verbose 1 -ia 1 -ib 1 -ja 1 -jb 1
Parameters: m=1 n=10 nrhs=1 mbA=2 nbA=2 mbB=2 nbB=2 mbQ=2 nbQ=2 mbZ=0 nbZ=0ia=1 ja=1 ib=1 jb=1 iq=1 jq=1 iz=0 jz=0 p=2 q=1 grid_layout= verbose=1
Parameters: m=1 n=10 nrhs=1 mbA=2 nbA=2 mbB=2 nbB=2 mbQ=2 nbQ=2 mbZ=0 nbZ=0ia=1 ja=1 ib=1 jb=1 iq=1 jq=1 iz=0 jz=0 p=2 q=1 grid_layout= verbose=1
[1719264274.005250] [c318-001:1866940:0]   ucc_proc_info.c:311  UCC  DEBUG proc pid 1866940, host c318-001, host_hash 2154568351744862994, sockid 0, numaid 0
[1719264274.005272] [c318-001:1866940:0] ucc_constructor.c:191  UCC  INFO  version: 1.3.0, loaded from: /scratch/projects/compilers/nvhpc_24.5/Linux_x86_64/24.5/comm_libs/12.4/hpcx/hpcx-2.19/ucc/lib/libucc.so.1, cfg file: /scratch/projects/compilers/nvhpc_24.5/Linux_x86_64/24.5/comm_libs/12.4/hpcx/hpcx-2.19/ucc/share/ucc.conf
[1719264274.005299] [c318-001:1866940:0]          ucc_mc.c:67   UCC  DEBUG mc cpu mc initialized
[1719264274.005354] [c318-001:1866939:0]   ucc_proc_info.c:311  UCC  DEBUG proc pid 1866939, host c318-001, host_hash 2154568351744862994, sockid 0, numaid 0
[1719264274.005369] [c318-001:1866939:0] ucc_constructor.c:191  UCC  INFO  version: 1.3.0, loaded from: /scratch/projects/compilers/nvhpc_24.5/Linux_x86_64/24.5/comm_libs/12.4/hpcx/hpcx-2.19/ucc/lib/libucc.so.1, cfg file: /scratch/projects/compilers/nvhpc_24.5/Linux_x86_64/24.5/comm_libs/12.4/hpcx/hpcx-2.19/ucc/share/ucc.conf
[1719264274.005392] [c318-001:1866939:0]          ucc_mc.c:67   UCC  DEBUG mc cpu mc initialized
[1719264274.015114] [c318-001:1866940:0]         mc_cuda.c:65   cuda mc DEBUG driver version 12020
[1719264274.015126] [c318-001:1866940:0]         mc_cuda.c:78   cuda mc DEBUG cuCtxGetDevice() failed: invalid device context
[1719264274.015133] [c318-001:1866940:0]          ucc_mc.c:67   UCC  DEBUG mc cuda mc initialized
[1719264274.015149] [c318-001:1866940:0]          ucc_ec.c:63   UCC  DEBUG ec cpu ec initialized
[1719264274.015610] [c318-001:1866939:0]         mc_cuda.c:65   cuda mc DEBUG driver version 12020
[1719264274.015618] [c318-001:1866939:0]         mc_cuda.c:78   cuda mc DEBUG cuCtxGetDevice() failed: invalid device context
[1719264274.015622] [c318-001:1866939:0]          ucc_mc.c:67   UCC  DEBUG mc cuda mc initialized
[1719264274.015635] [c318-001:1866939:0]          ucc_ec.c:63   UCC  DEBUG ec cpu ec initialized
[1719264274.019374] [c318-001:1866940:0]          ucc_ec.c:63   UCC  DEBUG ec cuda ec initialized
[1719264274.019411] [c318-001:1866940:0]    cl_basic_lib.c:20   CL_BASIC DEBUG initialized lib object: 0x13448b0
[1719264274.019423] [c318-001:1866940:0]         ucc_lib.c:152  UCC  DEBUG lib_prefix "OMPI_UCC_": initialized component "basic" score 10
[1719264274.019453] [c318-001:1866940:0]     cl_hier_lib.c:53   CL_HIER DEBUG initialized lib object: 0xfe2ac0
[1719264274.019458] [c318-001:1866940:0]         ucc_lib.c:152  UCC  DEBUG lib_prefix "OMPI_UCC_": initialized component "hier" score 50
[1719264274.019478] [c318-001:1866940:0]     tl_cuda_lib.c:35   TL_CUDA DEBUG initialized lib object: 0x12f9450
[1719264274.019491] [c318-001:1866940:0]     tl_self_lib.c:20   TL_SELF DEBUG initialized lib object: 0x12fbd90
[1719264274.019550] [c318-001:1866940:0]      tl_ucp_lib.c:69   TL_UCP DEBUG initialized lib object: 0xd23e20
[1719264274.019594] [c318-001:1866940:0] tl_cuda_context.c:43   TL_CUDA DEBUG cannot create CUDA TL context without active CUDA context
[1719264274.019601] [c318-001:1866940:0]     ucc_context.c:412  UCC  DEBUG failed to create tl context for cuda
[1719264274.020090] [c318-001:1866939:0]          ucc_ec.c:63   UCC  DEBUG ec cuda ec initialized
[1719264274.020125] [c318-001:1866939:0]    cl_basic_lib.c:20   CL_BASIC DEBUG initialized lib object: 0x2b46040
[1719264274.020133] [c318-001:1866939:0]         ucc_lib.c:152  UCC  DEBUG lib_prefix "OMPI_UCC_": initialized component "basic" score 10
[1719264274.020161] [c318-001:1866939:0]     cl_hier_lib.c:53   CL_HIER DEBUG initialized lib object: 0x27d5c20
[1719264274.020165] [c318-001:1866939:0]         ucc_lib.c:152  UCC  DEBUG lib_prefix "OMPI_UCC_": initialized component "hier" score 50
[1719264274.020182] [c318-001:1866939:0]     tl_cuda_lib.c:35   TL_CUDA DEBUG initialized lib object: 0x2ad4c30
[1719264274.020194] [c318-001:1866939:0]     tl_self_lib.c:20   TL_SELF DEBUG initialized lib object: 0x2b20130
[1719264274.020249] [c318-001:1866939:0]      tl_ucp_lib.c:69   TL_UCP DEBUG initialized lib object: 0x2516e20
[1719264274.020293] [c318-001:1866939:0] tl_cuda_context.c:43   TL_CUDA DEBUG cannot create CUDA TL context without active CUDA context
[1719264274.020299] [c318-001:1866939:0]     ucc_context.c:412  UCC  DEBUG failed to create tl context for cuda
[1719264274.058242] [c318-001:1866939:0]  tl_ucp_context.c:277  TL_UCP DEBUG initialized tl context: 0x2b0ebb0
[1719264274.058273] [c318-001:1866939:0] cl_basic_context.c:39   CL_BASIC DEBUG TL cuda context is not available, skipping
[1719264274.058277] [c318-001:1866939:0] cl_basic_context.c:50   CL_BASIC DEBUG initialized cl context: 0x2b512c0
[1719264274.058287] [c318-001:1866939:0] cl_hier_context.c:43   CL_HIER DEBUG TL cuda context is not available, skipping
[1719264274.058291] [c318-001:1866939:0] cl_hier_context.c:64   CL_HIER DEBUG initialized cl context: 0x2b4aad0
[1719264274.067381] [c318-001:1866940:0]  tl_ucp_context.c:277  TL_UCP DEBUG initialized tl context: 0x12f9850
[1719264274.067400] [c318-001:1866940:0] cl_basic_context.c:39   CL_BASIC DEBUG TL cuda context is not available, skipping
[1719264274.067404] [c318-001:1866940:0] cl_basic_context.c:50   CL_BASIC DEBUG initialized cl context: 0x1354f80
[1719264274.067412] [c318-001:1866940:0] cl_hier_context.c:43   CL_HIER DEBUG TL cuda context is not available, skipping
[1719264274.067416] [c318-001:1866940:0] cl_hier_context.c:64   CL_HIER DEBUG initialized cl context: 0x1349230
[1719264274.068571] [c318-001:1866939:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x2e107b0
[1719264274.068577] [c318-001:1866939:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x2e107b0
[1719264274.068580] [c318-001:1866939:0]     ucc_context.c:839  UCC  DEBUG created ucc context 0x2b47030 for lib OMPI_UCC_
[1719264274.068582] [c318-001:1866940:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x161c7a0
[1719264274.068588] [c318-001:1866940:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x161c7a0
[1719264274.068591] [c318-001:1866940:0]     ucc_context.c:839  UCC  DEBUG created ucc context 0x13458a0 for lib OMPI_UCC_
[1719264274.613824] [c318-001:1866940:0]        ucc_team.c:370  UCC  DEBUG team 0x1ab7610 rank 1, ctx_rank 1, map_type 1
[1719264274.613825] [c318-001:1866939:0]        ucc_team.c:370  UCC  DEBUG team 0x32a6230 rank 0, ctx_rank 0, map_type 1
[1719264274.613851] [c318-001:1866939:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264274.613859] [c318-001:1866939:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x3203d60
[1719264274.613851] [c318-001:1866940:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264274.613859] [c318-001:1866940:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x16c91f0
[1719264274.613881] [c318-001:1866939:0]     tl_shm_team.c:158  TL_SHM DEBUG using perf params: generic
[1719264274.622135] [c318-001:1866939:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264274.622147] [c318-001:1866939:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264274.622149] [c318-001:1866939:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x32b1de0
[1719264274.622151] [c318-001:1866939:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x32b1de0
[1719264274.622135] [c318-001:1866940:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264274.622147] [c318-001:1866940:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264274.622149] [c318-001:1866940:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x1ad0fc0
[1719264274.622151] [c318-001:1866940:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x1ad0fc0
[1719264274.622155] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264274.622158] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264274.622160] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264274.622155] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264274.622158] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264274.622160] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264274.622182] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264274.622184] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264274.622186] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264274.622182] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264274.622184] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264274.622185] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264274.622258] [c318-001:1866940:0]    cl_hier_team.c:58   CL_HIER DEBUG skipping single node team
[1719264274.622265] [c318-001:1866940:0]        ucc_team.c:297  UCC  DEBUG failed to create CL hier team
[1719264274.622258] [c318-001:1866939:0]    cl_hier_team.c:58   CL_HIER DEBUG skipping single node team
[1719264274.622265] [c318-001:1866939:0]        ucc_team.c:297  UCC  DEBUG failed to create CL hier team
[1719264274.622275] [c318-001:1866939:0]        ucc_team.c:472  UCC  INFO  ===== COLL_SCORE_MAP (team_id 32768, size 2) =====
[1719264274.622285] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgather:
[1719264274.622285] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264274.622285] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264274.622285] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264274.622296] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allreduce:
[1719264274.622296] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_SHM:10 {4K..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264274.622296] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264274.622296] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264274.622305] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoall:
[1719264274.622305] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..257}:TL_UCP:10 {258..inf}:TL_UCP:10
[1719264274.622305] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.622305] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.622319] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoallv:
[1719264274.622319] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264274.622319] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.622319] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.622332] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Barrier:
[1719264274.622332] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264274.622332] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.622332] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.622345] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Bcast:
[1719264274.622345] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264274.622345] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.622345] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.622361] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanin:
[1719264274.622361] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264274.622361] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.622361] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.622370] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanout:
[1719264274.622370] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264274.622370] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.622370] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.622382] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gather:
[1719264274.622382] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264274.622382] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.622382] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.622393] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gatherv:
[1719264274.622393] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264274.622393] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.622393] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.622405] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce:
[1719264274.622405] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264274.622405] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.622405] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.622416] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatter:
[1719264274.622416] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264274.622416] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.622416] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.622428] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatterv:
[1719264274.622428] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264274.622428] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.622428] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.622435] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatterv:
[1719264274.622435] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264274.622435] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.622435] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.622444] [c318-001:1866939:0]        ucc_team.c:474  UCC  INFO  ================================================
[1719264274.631501] [c318-001:1866939:0]        ucc_team.c:370  UCC  DEBUG team 0x33166e0 rank 0, ctx_rank 0, map_type 4
[1719264274.631511] [c318-001:1866939:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264274.631514] [c318-001:1866939:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x2e429b0
[1719264274.631522] [c318-001:1866939:0]     tl_shm_team.c:158  TL_SHM DEBUG using perf params: generic
[1719264274.631501] [c318-001:1866940:0]        ucc_team.c:370  UCC  DEBUG team 0x1b26830 rank 1, ctx_rank 1, map_type 4
[1719264274.631511] [c318-001:1866940:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264274.631513] [c318-001:1866940:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x17ae010
[1719264274.644294] [c318-001:1866940:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264274.644298] [c318-001:1866940:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264274.644300] [c318-001:1866940:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x1b26de0
[1719264274.644301] [c318-001:1866940:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x1b26de0
[1719264274.644304] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264274.644306] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264274.644308] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264274.644318] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264274.644320] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264274.644322] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264274.644363] [c318-001:1866940:0]    cl_hier_team.c:58   CL_HIER DEBUG skipping single node team
[1719264274.644366] [c318-001:1866940:0]        ucc_team.c:297  UCC  DEBUG failed to create CL hier team
[1719264274.644451] [c318-001:1866940:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0xe62990
[1719264274.644463] [c318-001:1866940:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x1b26de0
[1719264274.644474] [c318-001:1866940:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x17ae010
[1719264274.644536] [c318-001:1866939:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264274.644540] [c318-001:1866939:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264274.644542] [c318-001:1866939:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x3317080
[1719264274.644545] [c318-001:1866939:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x3317080
[1719264274.644548] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264274.644550] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264274.644553] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264274.644559] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264274.644562] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264274.644565] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264274.644599] [c318-001:1866939:0]    cl_hier_team.c:58   CL_HIER DEBUG skipping single node team
[1719264274.644602] [c318-001:1866939:0]        ucc_team.c:297  UCC  DEBUG failed to create CL hier team
[1719264274.644609] [c318-001:1866939:0]        ucc_team.c:472  UCC  INFO  ===== COLL_SCORE_MAP (team_id 32771, size 2) =====
[1719264274.644616] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgather:
[1719264274.644616] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264274.644616] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264274.644616] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264274.644629] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allreduce:
[1719264274.644629] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_SHM:10 {4K..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264274.644629] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264274.644629] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264274.644643] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoall:
[1719264274.644643] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..257}:TL_UCP:10 {258..inf}:TL_UCP:10
[1719264274.644643] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.644643] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.644652] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoallv:
[1719264274.644652] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264274.644652] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.644652] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.644659] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Barrier:
[1719264274.644659] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264274.644659] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.644659] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.644670] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Bcast:
[1719264274.644670] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264274.644670] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.644670] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.644676] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanin:
[1719264274.644676] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264274.644676] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.644676] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.644686] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanout:
[1719264274.644686] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264274.644686] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.644686] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.644697] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gather:
[1719264274.644697] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264274.644697] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.644697] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.644705] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gatherv:
[1719264274.644705] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264274.644705] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.644705] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.644718] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce:
[1719264274.644718] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264274.644718] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.644718] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.644726] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatter:
[1719264274.644726] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264274.644726] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.644726] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.644736] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatterv:
[1719264274.644736] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264274.644736] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.644736] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.644747] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatterv:
[1719264274.644747] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264274.644747] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264274.644747] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264274.644759] [c318-001:1866939:0]        ucc_team.c:474  UCC  INFO  ================================================
[1719264274.644838] [c318-001:1866939:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0x2655990
[1719264274.644847] [c318-001:1866939:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x3317080
[1719264274.644855] [c318-001:1866939:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x2e429b0
[1719264274.644859] [c318-001:1866939:0]        ucc_team.c:528  UCC  INFO  team destroyed, team_id 32771
[1719264275.064913] [c318-001:1866939:0]          ucc_mc.c:78   UCC  WARN  mc cpu mc was allready initilized with different thread mode: current tm 0, provided tm 2
[1719264275.064927] [c318-001:1866939:0]          ucc_mc.c:78   UCC  WARN  mc cuda mc was allready initilized with different thread mode: current tm 0, provided tm 2
[1719264275.064930] [c318-001:1866939:0]          ucc_ec.c:75   UCC  WARN  ec cpu ec was allready initilized with different thread mode: current tm 0, provided tm 2
[1719264275.064933] [c318-001:1866939:0]          ucc_ec.c:75   UCC  WARN  ec cuda ec was allready initilized with different thread mode: current tm 0, provided tm 2
[1719264275.064948] [c318-001:1866939:0]    cl_basic_lib.c:20   CL_BASIC DEBUG initialized lib object: 0x37479f0
[1719264275.064914] [c318-001:1866940:0]          ucc_mc.c:78   UCC  WARN  mc cpu mc was allready initilized with different thread mode: current tm 0, provided tm 2
[1719264275.064927] [c318-001:1866940:0]          ucc_mc.c:78   UCC  WARN  mc cuda mc was allready initilized with different thread mode: current tm 0, provided tm 2
[1719264275.064931] [c318-001:1866940:0]          ucc_ec.c:75   UCC  WARN  ec cpu ec was allready initilized with different thread mode: current tm 0, provided tm 2
[1719264275.064933] [c318-001:1866940:0]          ucc_ec.c:75   UCC  WARN  ec cuda ec was allready initilized with different thread mode: current tm 0, provided tm 2
[1719264275.064949] [c318-001:1866940:0]    cl_basic_lib.c:20   CL_BASIC DEBUG initialized lib object: 0x1f57f10
[1719264275.064956] [c318-001:1866940:0]         ucc_lib.c:152  UCC  DEBUG lib_prefix "CAL_UCC_": initialized component "basic" score 10
[1719264275.064956] [c318-001:1866939:0]         ucc_lib.c:152  UCC  DEBUG lib_prefix "CAL_UCC_": initialized component "basic" score 10
[1719264275.064971] [c318-001:1866939:0]     tl_cuda_lib.c:35   TL_CUDA DEBUG initialized lib object: 0x3c9eed0
[1719264275.064987] [c318-001:1866939:0]     tl_nccl_lib.c:16   TL_NCCL DEBUG initialized lib object: 0x3f5c3f0
[1719264275.064971] [c318-001:1866940:0]     tl_cuda_lib.c:35   TL_CUDA DEBUG initialized lib object: 0x24afb40
[1719264275.064987] [c318-001:1866940:0]     tl_nccl_lib.c:16   TL_NCCL DEBUG initialized lib object: 0x2739f40
[1719264275.064995] [c318-001:1866940:0]     tl_self_lib.c:20   TL_SELF DEBUG initialized lib object: 0x24999d0
[1719264275.064995] [c318-001:1866939:0]     tl_self_lib.c:20   TL_SELF DEBUG initialized lib object: 0x3c89030
[1719264275.065036] [c318-001:1866940:0]      tl_ucp_lib.c:69   TL_UCP DEBUG initialized lib object: 0x27661c0
[1719264275.065036] [c318-001:1866939:0]      tl_ucp_lib.c:69   TL_UCP DEBUG initialized lib object: 0x3f28e70
[1719264275.065066] [c318-001:1866939:0]     ucc_context.c:247  UCC  INFO  required TL sharp is not part of the context
[1719264275.065070] [c318-001:1866939:0]     ucc_context.c:247  UCC  INFO  required TL hcoll is not part of the context
[1719264275.065066] [c318-001:1866940:0]     ucc_context.c:247  UCC  INFO  required TL sharp is not part of the context
[1719264275.065070] [c318-001:1866940:0]     ucc_context.c:247  UCC  INFO  required TL hcoll is not part of the context
[1719264275.095681] [c318-001:1866940:0] tl_cuda_context.c:71   TL_CUDA DEBUG initialized tl context: 0x2152ef0
[1719264275.095698] [c318-001:1866940:0] tl_nccl_context.c:182  TL_NCCL DEBUG using memops completion sync
[1719264275.098553] [c318-001:1866940:0] tl_nccl_context.c:205  TL_NCCL DEBUG initialized tl context: 0x1f46270
[1719264275.098818] [c318-001:1866939:0] tl_cuda_context.c:71   TL_CUDA DEBUG initialized tl context: 0x39429d0
[1719264275.098832] [c318-001:1866939:0] tl_nccl_context.c:182  TL_NCCL DEBUG using memops completion sync
[1719264275.098948] [c318-001:1866939:0] tl_nccl_context.c:205  TL_NCCL DEBUG initialized tl context: 0x3736120
[1719264275.135466] [c318-001:1866939:0]  tl_ucp_context.c:277  TL_UCP DEBUG initialized tl context: 0x3712910
[1719264275.135478] [c318-001:1866939:0] cl_basic_context.c:50   CL_BASIC DEBUG initialized cl context: 0x3f68830
[1719264275.146446] [c318-001:1866940:0]  tl_ucp_context.c:277  TL_UCP DEBUG initialized tl context: 0x1f22a60
[1719264275.146458] [c318-001:1866940:0] cl_basic_context.c:50   CL_BASIC DEBUG initialized cl context: 0x2779590
[1719264275.146559] [c318-001:1866940:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x2d76470
[1719264275.146563] [c318-001:1866940:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x2d76470
[1719264275.146565] [c318-001:1866940:0]     ucc_context.c:839  UCC  DEBUG created ucc context 0x276dc70 for lib CAL_UCC_
[1719264275.146559] [c318-001:1866939:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x4376060
[1719264275.146563] [c318-001:1866939:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x4376060
[1719264275.146565] [c318-001:1866939:0]     ucc_context.c:839  UCC  DEBUG created ucc context 0x3f5cff0 for lib CAL_UCC_
[1719264275.182000] [c318-001:1866939:0]  tl_ucp_context.c:277  TL_UCP DEBUG initialized tl context: 0x41b7010
[1719264275.182013] [c318-001:1866939:0] cl_basic_context.c:39   CL_BASIC DEBUG TL cuda context is not available, skipping
[1719264275.182015] [c318-001:1866939:0] cl_basic_context.c:39   CL_BASIC DEBUG TL nccl context is not available, skipping
[1719264275.182017] [c318-001:1866939:0] cl_basic_context.c:50   CL_BASIC DEBUG initialized cl context: 0x41d6fb0
[1719264275.194596] [c318-001:1866940:0]  tl_ucp_context.c:277  TL_UCP DEBUG initialized tl context: 0x29c8010
[1719264275.194607] [c318-001:1866940:0] cl_basic_context.c:39   CL_BASIC DEBUG TL cuda context is not available, skipping
[1719264275.194609] [c318-001:1866940:0] cl_basic_context.c:39   CL_BASIC DEBUG TL nccl context is not available, skipping
[1719264275.194611] [c318-001:1866940:0] cl_basic_context.c:50   CL_BASIC DEBUG initialized cl context: 0x29e7c10
[1719264275.194652] [c318-001:1866940:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x340e050
[1719264275.194655] [c318-001:1866940:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x340e050
[1719264275.194657] [c318-001:1866940:0]     ucc_context.c:839  UCC  DEBUG created ucc context 0x2d76940 for lib CAL_UCC_
[1719264275.194652] [c318-001:1866939:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x4b62ac0
[1719264275.194656] [c318-001:1866939:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x4b62ac0
[1719264275.194658] [c318-001:1866939:0]     ucc_context.c:839  UCC  DEBUG created ucc context 0x456dd30 for lib CAL_UCC_
[1719264275.194676] [c318-001:1866939:0]        ucc_team.c:370  UCC  DEBUG team 0x4c23050 rank 0, ctx_rank 0, map_type 3
[1719264275.194678] [c318-001:1866940:0]        ucc_team.c:370  UCC  DEBUG team 0x340e520 rank 1, ctx_rank 1, map_type 3
[1719264275.202804] [c318-001:1866940:0]    tl_cuda_team.c:109  TL_CUDA DEBUG posted tl team: 0x3438050
[1719264275.202814] [c318-001:1866940:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x1f4fde0
[1719264275.202842] [c318-001:1866940:0] tl_cuda_team_topo.c:315  TL_CUDA DEBUG no proxy found between dev 0000:21:00.0 (0) and dev 0000:e1:00.0 (1), cuda topology is not supported
[1719264275.202849] [c318-001:1866940:0]    tl_cuda_team.c:134  TL_CUDA DEBUG finalizing tl team: 0x3438050
[1719264275.203302] [c318-001:1866939:0]    tl_cuda_team.c:109  TL_CUDA DEBUG posted tl team: 0x4c23880
[1719264275.203310] [c318-001:1866939:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x373fc20
[1719264275.203329] [c318-001:1866939:0] tl_cuda_team_topo.c:315  TL_CUDA DEBUG no proxy found between dev 0000:21:00.0 (0) and dev 0000:e1:00.0 (1), cuda topology is not supported
[1719264275.203335] [c318-001:1866939:0]    tl_cuda_team.c:134  TL_CUDA DEBUG finalizing tl team: 0x4c23880
[1719264275.290771] [c318-001:1866940:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264275.290771] [c318-001:1866939:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264275.290790] [c318-001:1866939:0]     tl_shm_team.c:158  TL_SHM DEBUG using perf params: generic
[1719264275.290845] [c318-001:1866940:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264275.290849] [c318-001:1866940:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264275.290851] [c318-001:1866940:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x34ff090
[1719264275.290853] [c318-001:1866940:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x34ff090
[1719264275.290855] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl cuda team: (-1)
[1719264275.290858] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl nccl team
[1719264275.290845] [c318-001:1866939:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264275.290849] [c318-001:1866939:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264275.290851] [c318-001:1866939:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x4cea090
[1719264275.290853] [c318-001:1866939:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x4cea090
[1719264275.290855] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl cuda team: (-1)
[1719264275.290857] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl nccl team
[1719264275.290859] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264275.290861] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264275.290862] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264275.290860] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264275.290862] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264275.290865] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264275.290893] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264275.290896] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264275.290898] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264275.290893] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264275.290896] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264275.290898] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264275.290941] [c318-001:1866939:0]        ucc_team.c:472  UCC  INFO  ===== COLL_SCORE_MAP (team_id 32768, size 2) =====
[1719264275.290951] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgather:
[1719264275.290951] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264275.290951] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_NCCL:10 {4K..inf}:TL_NCCL:10
[1719264275.290951] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_NCCL:10 {4K..inf}:TL_NCCL:10
[1719264275.290963] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgatherv:
[1719264275.290963] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.290963] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..16383}:TL_NCCL:10 {16K..1048575}:TL_NCCL:10 {1M..inf}:TL_NCCL:10
[1719264275.290963] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.290975] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allreduce:
[1719264275.290975] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_SHM:10 {4K..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.290975] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_NCCL:10 {4K..inf}:TL_NCCL:10
[1719264275.290975] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_NCCL:10 {4K..inf}:TL_NCCL:10
[1719264275.290986] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoall:
[1719264275.290986] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..257}:TL_UCP:10 {258..inf}:TL_UCP:10
[1719264275.290986] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.290986] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.290997] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoallv:
[1719264275.290997] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.290997] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.290997] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.291008] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Barrier:
[1719264275.291008] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.291008] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.291008] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.291015] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Bcast:
[1719264275.291015] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.291015] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.291015] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.291030] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanin:
[1719264275.291030] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.291030] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.291030] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.291043] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanout:
[1719264275.291043] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.291043] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.291043] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.291050] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gather:
[1719264275.291050] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.291050] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.291050] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.291062] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gatherv:
[1719264275.291062] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.291062] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.291062] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.291075] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce:
[1719264275.291075] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.291075] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.291075] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.291089] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatter:
[1719264275.291089] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.291089] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.291089] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.291100] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatterv:
[1719264275.291100] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.291100] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.291100] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.291113] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatter:
[1719264275.291113] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.291113] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.291122] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatterv:
[1719264275.291122] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.291122] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.291122] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.291133] [c318-001:1866939:0]        ucc_team.c:474  UCC  INFO  ================================================
print_host_matrix : Input matrix A
11.72  0.39  0.78  0.80  0.91  0.20  0.34  0.77  0.28  0.55
0.39  10.69  0.63  0.36  0.51  0.95  0.92  0.64  0.72  0.14
0.78  0.63  4.99  0.02  0.24  0.14  0.80  0.16  0.40  0.13
0.80  0.36  0.02  7.17  1.00  0.22  0.51  0.84  0.61  0.30
0.91  0.51  0.24  1.00  7.38  0.52  0.49  0.97  0.29  0.77
0.20  0.95  0.14  0.22  0.52  5.74  0.77  0.40  0.89  0.28
0.34  0.92  0.80  0.51  0.49  0.77  4.30  0.81  0.92  0.07
0.77  0.64  0.16  0.84  0.97  0.40  0.81  3.12  0.53  0.09
0.28  0.72  0.40  0.61  0.29  0.89  0.92  0.53  1.71  0.66
0.55  0.14  0.13  0.30  0.77  0.28  0.07  0.09  0.66  1.78
print_host_matrix : Input matrix X
-nan
-nan
-nan
-nan
-nan
-nan
-nan
-nan
-nan
-nan
print_host_matrix : Input matrix B
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
[1719264275.680628] [c318-001:1866940:0]        ucc_team.c:370  UCC  DEBUG team 0x3c55480 rank 1, ctx_rank 1, map_type 3
[1719264275.680648] [c318-001:1866940:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264275.680652] [c318-001:1866940:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x17ae010
[1719264275.680650] [c318-001:1866939:0]        ucc_team.c:370  UCC  DEBUG team 0x5443ed0 rank 0, ctx_rank 0, map_type 3
[1719264275.680661] [c318-001:1866939:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264275.680665] [c318-001:1866939:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x3689cc0
[1719264275.680674] [c318-001:1866939:0]     tl_shm_team.c:158  TL_SHM DEBUG using perf params: generic
[1719264275.689127] [c318-001:1866939:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264275.689137] [c318-001:1866939:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264275.689140] [c318-001:1866939:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x544f840
[1719264275.689141] [c318-001:1866939:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x544f840
[1719264275.689127] [c318-001:1866940:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264275.689137] [c318-001:1866940:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264275.689140] [c318-001:1866940:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x3ca6b10
[1719264275.689142] [c318-001:1866940:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x3ca6b10
[1719264275.689145] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264275.689148] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264275.689149] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264275.689152] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264275.689153] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264275.689155] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264275.689145] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264275.689148] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264275.689149] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264275.689152] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264275.689154] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264275.689155] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264275.689200] [c318-001:1866939:0]        ucc_team.c:472  UCC  INFO  ===== COLL_SCORE_MAP (team_id 32769, size 2) =====
[1719264275.689209] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgather:
[1719264275.689209] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264275.689209] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264275.689209] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264275.689220] [c318-001:1866940:0] ucc_coll_score_map.c:144  UCC  DEBUG coll Bcast is not supported for TL_SHM, fallback TL_UCP
[1719264275.689221] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgatherv:
[1719264275.689221] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.689221] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.689221] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.689231] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allreduce:
[1719264275.689231] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_SHM:10 {4K..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.689231] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264275.689231] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264275.689245] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoall:
[1719264275.689245] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..257}:TL_UCP:10 {258..inf}:TL_UCP:10
[1719264275.689245] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.689245] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.689263] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoallv:
[1719264275.689263] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.689263] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.689263] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.689272] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Barrier:
[1719264275.689272] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.689272] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.689272] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.689283] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Bcast:
[1719264275.689283] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.689283] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.689283] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.689293] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanin:
[1719264275.689293] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.689293] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.689293] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.689306] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanout:
[1719264275.689306] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.689306] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.689306] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.689319] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gather:
[1719264275.689319] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.689319] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.689319] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.689328] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gatherv:
[1719264275.689328] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.689328] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.689328] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.689339] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce:
[1719264275.689339] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.689339] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.689339] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.689354] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatter:
[1719264275.689354] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.689354] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.689354] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.689366] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatterv:
[1719264275.689366] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.689366] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.689366] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.689379] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatterv:
[1719264275.689379] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.689379] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.689379] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.689388] [c318-001:1866939:0]        ucc_team.c:474  UCC  INFO  ================================================
[1719264275.689592] [c318-001:1866939:0] ucc_coll_score_map.c:144  UCC  DEBUG coll Bcast is not supported for TL_SHM, fallback TL_UCP
[1719264275.689641] [c318-001:1866939:0] ucc_coll_score_map.c:144  UCC  DEBUG coll Bcast is not supported for TL_SHM, fallback TL_UCP
[1719264275.689663] [c318-001:1866940:0] ucc_coll_score_map.c:144  UCC  DEBUG coll Bcast is not supported for TL_SHM, fallback TL_UCP
[1719264275.689697] [c318-001:1866940:0]        ucc_team.c:370  UCC  DEBUG team 0x3c64e90 rank 1, ctx_rank 1, map_type 3
[1719264275.689697] [c318-001:1866939:0]        ucc_team.c:370  UCC  DEBUG team 0x5453fa0 rank 0, ctx_rank 0, map_type 3
[1719264275.689944] [c318-001:1866940:0]    tl_cuda_team.c:109  TL_CUDA DEBUG posted tl team: 0x3c65930
[1719264275.689948] [c318-001:1866940:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x164ec60
[1719264275.689955] [c318-001:1866939:0]    tl_cuda_team.c:109  TL_CUDA DEBUG posted tl team: 0x5454a40
[1719264275.689959] [c318-001:1866939:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x2e429b0
[1719264275.689963] [c318-001:1866939:0] tl_cuda_team_topo.c:315  TL_CUDA DEBUG no proxy found between dev 0000:21:00.0 (0) and dev 0000:e1:00.0 (1), cuda topology is not supported
[1719264275.689957] [c318-001:1866940:0] tl_cuda_team_topo.c:315  TL_CUDA DEBUG no proxy found between dev 0000:21:00.0 (0) and dev 0000:e1:00.0 (1), cuda topology is not supported
[1719264275.689960] [c318-001:1866940:0]    tl_cuda_team.c:134  TL_CUDA DEBUG finalizing tl team: 0x3c65930
[1719264275.689966] [c318-001:1866939:0]    tl_cuda_team.c:134  TL_CUDA DEBUG finalizing tl team: 0x5454a40
[1719264275.690245] [c318-001:1866940:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264275.690247] [c318-001:1866939:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264275.690257] [c318-001:1866939:0]     tl_shm_team.c:158  TL_SHM DEBUG using perf params: generic
[1719264275.690291] [c318-001:1866939:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264275.690294] [c318-001:1866939:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264275.690296] [c318-001:1866939:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x54ad360
[1719264275.690298] [c318-001:1866939:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x54ad360
[1719264275.690291] [c318-001:1866940:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264275.690294] [c318-001:1866940:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264275.690296] [c318-001:1866940:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x3c66850
[1719264275.690297] [c318-001:1866940:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x3c66850
[1719264275.690300] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl cuda team: (-1)
[1719264275.690303] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl nccl team
[1719264275.690305] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264275.690308] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264275.690310] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264275.690301] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl cuda team: (-1)
[1719264275.690305] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl nccl team
[1719264275.690306] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264275.690310] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264275.690312] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264275.690325] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264275.690328] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264275.690330] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264275.690326] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264275.690330] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264275.690332] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264275.690371] [c318-001:1866939:0]        ucc_team.c:472  UCC  INFO  ===== COLL_SCORE_MAP (team_id 32770, size 2) =====
[1719264275.690378] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgather:
[1719264275.690378] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264275.690378] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_NCCL:10 {4K..inf}:TL_NCCL:10
[1719264275.690378] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_NCCL:10 {4K..inf}:TL_NCCL:10
[1719264275.690390] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgatherv:
[1719264275.690390] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.690390] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..16383}:TL_NCCL:10 {16K..1048575}:TL_NCCL:10 {1M..inf}:TL_NCCL:10
[1719264275.690390] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.690399] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allreduce:
[1719264275.690399] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_SHM:10 {4K..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.690399] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_NCCL:10 {4K..inf}:TL_NCCL:10
[1719264275.690399] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_NCCL:10 {4K..inf}:TL_NCCL:10
[1719264275.690409] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoall:
[1719264275.690409] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..257}:TL_UCP:10 {258..inf}:TL_UCP:10
[1719264275.690409] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.690409] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.690417] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoallv:
[1719264275.690417] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.690417] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.690417] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.690428] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Barrier:
[1719264275.690428] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.690428] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.690428] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.690437] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Bcast:
[1719264275.690437] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.690437] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.690437] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.690449] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanin:
[1719264275.690449] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.690449] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.690449] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.690460] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanout:
[1719264275.690460] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.690460] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.690460] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.690469] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gather:
[1719264275.690469] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.690469] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.690469] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.690478] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gatherv:
[1719264275.690478] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.690478] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.690478] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.690489] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce:
[1719264275.690489] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.690489] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.690489] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.690498] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatter:
[1719264275.690498] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.690498] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.690498] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.690514] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatterv:
[1719264275.690514] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.690514] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.690514] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.690526] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatter:
[1719264275.690526] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.690526] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.690534] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatterv:
[1719264275.690534] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.690534] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.690534] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.690542] [c318-001:1866939:0]        ucc_team.c:474  UCC  INFO  ================================================
[1719264275.690562] [c318-001:1866939:0]    tl_self_team.c:19   TL_SELF DEBUG posted tl team: 0x3f6b610
[1719264275.690565] [c318-001:1866939:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x2627e50
[1719264275.690567] [c318-001:1866939:0]    tl_self_team.c:42   TL_SELF DEBUG initialized tl team: 0x3f6b610
[1719264275.690569] [c318-001:1866939:0]          ucc_tl.c:293  TL_SHM DEBUG team size 1 is too small, min supported 2
[1719264275.690563] [c318-001:1866940:0]    tl_self_team.c:19   TL_SELF DEBUG posted tl team: 0x277d150
[1719264275.690566] [c318-001:1866940:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0xe34e50
[1719264275.690568] [c318-001:1866940:0]    tl_self_team.c:42   TL_SELF DEBUG initialized tl team: 0x277d150
[1719264275.690570] [c318-001:1866940:0]          ucc_tl.c:293  TL_SHM DEBUG team size 1 is too small, min supported 2
[1719264275.690575] [c318-001:1866940:0]          ucc_tl.c:293  TL_UCP DEBUG team size 1 is too small, min supported 2
[1719264275.690577] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl self team
[1719264275.690579] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl shm team: (-1)
[1719264275.690571] [c318-001:1866939:0]          ucc_tl.c:293  TL_UCP DEBUG team size 1 is too small, min supported 2
[1719264275.690576] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl self team
[1719264275.690578] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl shm team: (-1)
[1719264275.690580] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl ucp team: (-1)
[1719264275.690588] [c318-001:1866939:0]        ucc_team.c:472  UCC  INFO  ===== COLL_SCORE_MAP (team_id 32771, size 1) =====
[1719264275.690593] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgather:
[1719264275.690582] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl ucp team: (-1)
[1719264275.690591] [c318-001:1866940:0]        ucc_team.c:472  UCC  INFO  ===== COLL_SCORE_MAP (team_id 32771, size 1) =====
[1719264275.690593] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690593] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690593] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690593] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690593] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690607] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgatherv:
[1719264275.690607] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690607] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690607] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690607] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690607] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690622] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allreduce:
[1719264275.690622] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690601] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Allgather:
[1719264275.690601] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690601] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690601] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690601] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690601] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690623] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Allgatherv:
[1719264275.690623] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690623] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690623] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690622] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690622] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690622] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690622] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690650] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoall:
[1719264275.690650] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690650] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690650] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690650] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690650] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690674] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoallv:
[1719264275.690674] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690674] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690674] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690674] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690674] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690623] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690623] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690652] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Allreduce:
[1719264275.690652] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690652] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690652] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690652] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690652] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690679] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoall:
[1719264275.690679] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690679] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690679] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690679] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690679] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690703] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoallv:
[1719264275.690703] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690703] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690695] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Barrier:
[1719264275.690695] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690695] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690695] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690695] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690695] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690721] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Bcast:
[1719264275.690721] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690721] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690721] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690721] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690721] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690740] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanin:
[1719264275.690740] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690740] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690740] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690740] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690740] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690758] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanout:
[1719264275.690758] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690758] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  [1719264275.690703] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO      CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690703] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690703] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690725] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Barrier:
[1719264275.690725] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690725] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690725] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690725] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690725] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690755] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Bcast:
[1719264275.690755] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690755] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690755] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690755] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
        Cuda: {0..inf}:TL_SELF:10
[1719264275.690758] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690758] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690758] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690774] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gather:
[1719264275.690774] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690774] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690774] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690774] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690774] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690808] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gatherv:
[1719264275.690808] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690808] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690808] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690808] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690808] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690831] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce:
[1719264275.690831] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690831] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690831] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690831] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690831] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690861] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatter:
[1719264275.690755] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690784] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Fanin:
[1719264275.690784] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690784] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690784] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690784] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690784] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690810] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Fanout:
[1719264275.690810] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690810] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690810] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690810] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690810] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690835] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Gather:
[1719264275.690835] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690835] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690835] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690835] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690835] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690864] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Gatherv:
[1719264275.690864] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690864] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690864] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  [1719264275.690861] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO      Host: {0..inf}:TL_SELF:10
[1719264275.690861] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690861] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690861] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690861] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690890] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatterv:
[1719264275.690890] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690890] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690890] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690890] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690890] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690922] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatter:
[1719264275.690922] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690922] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690922] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690922] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690922] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690937] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatterv:
[1719264275.690937] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690937] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690937] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690937] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
        CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690864] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690864] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690894] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce:
[1719264275.690894] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690894] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690894] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690894] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690894] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690916] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatter:
[1719264275.690916] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690916] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690916] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690916] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690916] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690937] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatterv:
[1719264275.690937] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690937] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690937] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690937] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690937] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690962] [c318-001:1866939:0]        ucc_team.c:474  UCC  INFO  ================================================
[1719264275.691029] [c318-001:1866939:0]        ucc_team.c:370  UCC  DEBUG team 0x54b6190 rank 0, ctx_rank 0, map_type 3
[1719264275.690937] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690967] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Scatter:
[1719264275.690967] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690967] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690967] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690967] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690967] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.690999] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Scatterv:
[1719264275.690999] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.690999] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.690999] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.690999] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.690999] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691019] [c318-001:1866940:0]        ucc_team.c:474  UCC  INFO  ================================================
[1719264275.691029] [c318-001:1866940:0]        ucc_team.c:370  UCC  DEBUG team 0x3cc7570 rank 1, ctx_rank 1, map_type 3
[1719264275.691157] [c318-001:1866940:0]    tl_cuda_team.c:109  TL_CUDA DEBUG posted tl team: 0x3cc8010
[1719264275.691161] [c318-001:1866940:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x948190
[1719264275.691173] [c318-001:1866939:0]    tl_cuda_team.c:109  TL_CUDA DEBUG posted tl team: 0x54b6c30
[1719264275.691178] [c318-001:1866939:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x213b190
[1719264275.691175] [c318-001:1866940:0] tl_cuda_team_topo.c:315  TL_CUDA DEBUG no proxy found between dev 0000:21:00.0 (0) and dev 0000:e1:00.0 (1), cuda topology is not supported
[1719264275.691179] [c318-001:1866940:0]    tl_cuda_team.c:134  TL_CUDA DEBUG finalizing tl team: 0x3cc8010
[1719264275.691183] [c318-001:1866939:0] tl_cuda_team_topo.c:315  TL_CUDA DEBUG no proxy found between dev 0000:21:00.0 (0) and dev 0000:e1:00.0 (1), cuda topology is not supported
[1719264275.691188] [c318-001:1866939:0]    tl_cuda_team.c:134  TL_CUDA DEBUG finalizing tl team: 0x54b6c30
[1719264275.691391] [c318-001:1866940:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264275.691392] [c318-001:1866939:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264275.691398] [c318-001:1866939:0]     tl_shm_team.c:158  TL_SHM DEBUG using perf params: generic
[1719264275.691429] [c318-001:1866940:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264275.691432] [c318-001:1866940:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264275.691434] [c318-001:1866940:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x3cc8f50
[1719264275.691436] [c318-001:1866940:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x3cc8f50
[1719264275.691438] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl cuda team: (-1)
[1719264275.691429] [c318-001:1866939:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264275.691432] [c318-001:1866939:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264275.691433] [c318-001:1866939:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x54b7b70
[1719264275.691435] [c318-001:1866939:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x54b7b70
[1719264275.691436] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl cuda team: (-1)
[1719264275.691438] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl nccl team
[1719264275.691440] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl nccl team
[1719264275.691443] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264275.691445] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264275.691446] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264275.691440] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264275.691443] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264275.691444] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264275.691458] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264275.691460] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264275.691463] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264275.691459] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264275.691461] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264275.691463] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264275.691501] [c318-001:1866939:0]        ucc_team.c:472  UCC  INFO  ===== COLL_SCORE_MAP (team_id 32772, size 2) =====
[1719264275.691507] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgather:
[1719264275.691507] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264275.691507] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_NCCL:10 {4K..inf}:TL_NCCL:10
[1719264275.691507] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_NCCL:10 {4K..inf}:TL_NCCL:10
[1719264275.691520] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgatherv:
[1719264275.691520] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.691520] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..16383}:TL_NCCL:10 {16K..1048575}:TL_NCCL:10 {1M..inf}:TL_NCCL:10
[1719264275.691520] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.691533] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allreduce:
[1719264275.691533] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_SHM:10 {4K..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.691533] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_NCCL:10 {4K..inf}:TL_NCCL:10
[1719264275.691533] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_NCCL:10 {4K..inf}:TL_NCCL:10
[1719264275.691548] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoall:
[1719264275.691548] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..257}:TL_UCP:10 {258..inf}:TL_UCP:10
[1719264275.691548] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.691548] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.691555] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoallv:
[1719264275.691555] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.691555] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.691555] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.691565] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Barrier:
[1719264275.691565] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.691565] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.691565] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.691576] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Bcast:
[1719264275.691576] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.691576] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.691576] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.691584] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanin:
[1719264275.691584] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.691584] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.691584] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.691595] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanout:
[1719264275.691595] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.691595] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.691595] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.691607] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gather:
[1719264275.691607] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.691607] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.691607] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.691618] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gatherv:
[1719264275.691618] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.691618] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.691618] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.691636] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce:
[1719264275.691636] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.691636] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.691636] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.691652] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatter:
[1719264275.691652] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.691652] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.691652] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.691664] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatterv:
[1719264275.691664] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.691664] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.691664] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.691672] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatter:
[1719264275.691672] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.691672] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.691677] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatterv:
[1719264275.691677] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.691677] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_NCCL:10
[1719264275.691677] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_NCCL:10
[1719264275.691690] [c318-001:1866939:0]        ucc_team.c:474  UCC  INFO  ================================================
[1719264275.691706] [c318-001:1866940:0]    tl_self_team.c:19   TL_SELF DEBUG posted tl team: 0x1352780
[1719264275.691708] [c318-001:1866940:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x3cc9aa0
[1719264275.691710] [c318-001:1866940:0]    tl_self_team.c:42   TL_SELF DEBUG initialized tl team: 0x1352780
[1719264275.691713] [c318-001:1866940:0]          ucc_tl.c:293  TL_SHM DEBUG team size 1 is too small, min supported 2
[1719264275.691717] [c318-001:1866940:0]          ucc_tl.c:293  TL_UCP DEBUG team size 1 is too small, min supported 2
[1719264275.691719] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl self team
[1719264275.691721] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl shm team: (-1)
[1719264275.691724] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl ucp team: (-1)
[1719264275.691735] [c318-001:1866940:0]        ucc_team.c:472  UCC  INFO  ===== COLL_SCORE_MAP (team_id 32773, size 1) =====
[1719264275.691740] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Allgather:
[1719264275.691740] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691740] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691740] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691740] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691740] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691759] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Allgatherv:
[1719264275.691759] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691759] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691759] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691759] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691759] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691769] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Allreduce:
[1719264275.691769] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691769] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691769] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691769] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691769] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691783] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoall:
[1719264275.691783] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691783] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691783] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691783] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691783] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691800] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoallv:
[1719264275.691800] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691800] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691800] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691800] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691800] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691815] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Barrier:
[1719264275.691815] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691815] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691815] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691815] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691815] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691837] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Bcast:
[1719264275.691837] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691837] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691837] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691837] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691837] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691862] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Fanin:
[1719264275.691862] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691862] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691862] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691862] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691862] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691877] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Fanout:
[1719264275.691877] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691877] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691877] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691877] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691877] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691900] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Gather:
[1719264275.691900] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691900] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691900] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691900] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691900] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691917] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Gatherv:
[1719264275.691917] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691917] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691917] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691917] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691917] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691935] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce:
[1719264275.691935] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691935] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691935] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691935] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691935] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691949] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatter:
[1719264275.691949] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691949] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691949] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691949] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691949] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  [1719264275.691958] [c318-001:1866939:0]    tl_self_team.c:19   TL_SELF DEBUG posted tl team: 0x3f6b4a0
[1719264275.691965] [c318-001:1866939:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x54b89a0
[1719264275.691972] [c318-001:1866939:0]    tl_self_team.c:42   TL_SELF DEBUG initialized tl team: 0x3f6b4a0
[1719264275.691975] [c318-001:1866939:0]          ucc_tl.c:293  TL_SHM DEBUG team size 1 is too small, min supported 2
[1719264275.691978] [c318-001:1866939:0]          ucc_tl.c:293  TL_UCP DEBUG team size 1 is too small, min supported 2
[1719264275.691981] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl self team
        RocmManaged: {0..inf}:TL_SELF:10
[1719264275.691979] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatterv:
[1719264275.691979] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.691979] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.691979] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.691979] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.691979] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692000] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Scatter:
[1719264275.692000] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692000] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  [1719264275.691984] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl shm team: (-1)
[1719264275.691989] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl ucp team: (-1)
[1719264275.691996] [c318-001:1866939:0]        ucc_team.c:472  UCC  INFO  ===== COLL_SCORE_MAP (team_id 32773, size 1) =====
[1719264275.692002] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgather:
[1719264275.692002] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
        Cuda: {0..inf}:TL_SELF:10
[1719264275.692000] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692000] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692000] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692030] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO  Scatterv:
[1719264275.692030] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692030] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692030] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692030] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692030] [c318-001:1866940:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692049] [c318-001:1866940:0]        ucc_team.c:474  UCC  INFO  ================================================
[1719264275.692002] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692002] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692002] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692002] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692039] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgatherv:
[1719264275.692039] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692039] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692039] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692039] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692039] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692057] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allreduce:
[1719264275.692057] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692057] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692057] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692057] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692057] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692074] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoall:
[1719264275.692074] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692074] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692074] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692074] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692074] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692097] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoallv:
[1719264275.692097] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692097] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692097] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692097] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692097] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692114] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Barrier:
[1719264275.692114] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692114] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692114] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692114] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692114] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692130] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Bcast:
[1719264275.692130] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692130] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692130] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692130] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692130] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692151] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanin:
[1719264275.692151] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692151] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692151] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692151] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692151] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692165] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanout:
[1719264275.692165] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692165] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692165] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692165] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692165] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692179] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gather:
[1719264275.692179] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692179] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692179] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692179] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692179] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692198] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gatherv:
[1719264275.692198] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692198] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692198] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692198] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692198] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692216] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce:
[1719264275.692216] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692216] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692216] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692216] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692216] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692234] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatter:
[1719264275.692234] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692234] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692234] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692234] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692234] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692245] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatterv:
[1719264275.692245] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692245] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692245] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692245] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692245] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692262] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatter:
[1719264275.692262] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692262] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692262] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692262] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692262] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692278] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatterv:
[1719264275.692278] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SELF:10
[1719264275.692278] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_SELF:10
[1719264275.692278] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_SELF:10
[1719264275.692278] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Rocm: {0..inf}:TL_SELF:10
[1719264275.692278] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    RocmManaged: {0..inf}:TL_SELF:10
[1719264275.692296] [c318-001:1866939:0]        ucc_team.c:474  UCC  INFO  ================================================
[1719264275.692309] [c318-001:1866940:0]        ucc_team.c:370  UCC  DEBUG team 0x3cd1ea0 rank 1, ctx_rank 1, map_type 3
[1719264275.692312] [c318-001:1866940:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264275.692313] [c318-001:1866939:0]        ucc_team.c:370  UCC  DEBUG team 0x54c0df0 rank 0, ctx_rank 0, map_type 3
[1719264275.692316] [c318-001:1866939:0]          ucc_tl.c:299  TL_SELF DEBUG team size 2 is too big, max supported 1
[1719264275.692318] [c318-001:1866939:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x54c1260
[1719264275.692322] [c318-001:1866939:0]     tl_shm_team.c:158  TL_SHM DEBUG using perf params: generic
[1719264275.692314] [c318-001:1866940:0]   cl_basic_team.c:52   CL_BASIC DEBUG posted cl team: 0x3cd2310
[1719264275.692355] [c318-001:1866940:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264275.692358] [c318-001:1866940:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264275.692359] [c318-001:1866940:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x3cd27b0
[1719264275.692360] [c318-001:1866940:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x3cd27b0
[1719264275.692355] [c318-001:1866939:0]     tl_ucp_team.c:84   UCC  DEBUG section not found
[1719264275.692358] [c318-001:1866939:0]     tl_ucp_team.c:100  TL_UCP DEBUG opt knomial radix: 2
[1719264275.692359] [c318-001:1866939:0]     tl_ucp_team.c:103  TL_UCP DEBUG posted tl team: 0x54c1700
[1719264275.692361] [c318-001:1866939:0]     tl_ucp_team.c:202  TL_UCP DEBUG initialized tl team: 0x54c1700
[1719264275.692363] [c318-001:1866939:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264275.692366] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264275.692368] [c318-001:1866939:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264275.692363] [c318-001:1866940:0]   cl_basic_team.c:126  CL_BASIC DEBUG failed to create tl self team: (-1)
[1719264275.692367] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl shm team
[1719264275.692370] [c318-001:1866940:0]   cl_basic_team.c:122  CL_BASIC DEBUG initialized tl ucp team
[1719264275.692374] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264275.692376] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264275.692377] [c318-001:1866940:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264275.692372] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type host
[1719264275.692375] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda
[1719264275.692377] [c318-001:1866939:0]     tl_ucp_team.c:232  TL_UCP DEBUG enable support for memory type cuda-managed
[1719264275.692412] [c318-001:1866939:0]        ucc_team.c:472  UCC  INFO  ===== COLL_SCORE_MAP (team_id 32774, size 2) =====
[1719264275.692418] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgather:
[1719264275.692418] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264275.692418] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264275.692418] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264275.692427] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allgatherv:
[1719264275.692427] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.692427] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.692427] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.692441] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Allreduce:
[1719264275.692441] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..4095}:TL_SHM:10 {4K..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.692441] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264275.692441] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..4095}:TL_UCP:10 {4K..inf}:TL_UCP:10
[1719264275.692450] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoall:
[1719264275.692450] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..257}:TL_UCP:10 {258..inf}:TL_UCP:10
[1719264275.692450] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.692450] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.692461] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Alltoallv:
[1719264275.692461] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.692461] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.692461] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.692470] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Barrier:
[1719264275.692470] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.692470] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.692470] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.692482] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Bcast:
[1719264275.692482] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.692482] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.692482] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.692495] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanin:
[1719264275.692495] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.692495] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.692495] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.692508] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Fanout:
[1719264275.692508] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_SHM:10
[1719264275.692508] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.692508] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.692520] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gather:
[1719264275.692520] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.692520] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.692520] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.692530] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Gatherv:
[1719264275.692530] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.692530] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.692530] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.692544] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce:
[1719264275.692544] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..8K}:TL_SHM:10 {8193..inf}:TL_UCP:10
[1719264275.692544] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.692544] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.692557] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatter:
[1719264275.692557] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.692557] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.692557] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.692571] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Reduce_scatterv:
[1719264275.692571] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.692571] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.692571] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.692579] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO  Scatterv:
[1719264275.692579] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Host: {0..inf}:TL_UCP:10
[1719264275.692579] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    Cuda: {0..inf}:TL_UCP:10
[1719264275.692579] [c318-001:1866939:0] ucc_coll_score_map.c:206  UCC  INFO    CudaManaged: {0..inf}:TL_UCP:10
[1719264275.692591] [c318-001:1866939:0]        ucc_team.c:474  UCC  INFO  ================================================
[1719264276.679172] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264276.679175] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264276.680274] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264276.680277] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264276.886054] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264276.886053] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264276.886130] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264276.886133] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.049606] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.049610] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.049668] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.049671] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.212885] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.212899] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.212956] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.212959] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.213000] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.213042] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.213053] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.213277] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.213321] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.213339] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.213383] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.213385] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.213392] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.217096] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.217142] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.217144] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.217169] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.217170] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.217200] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.217204] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.217234] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.217238] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.217277] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.217279] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.217294] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.217312] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.217355] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.217361] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.217357] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.217370] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.217415] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.217417] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.217420] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.217444] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.217586] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.217588] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.217601] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.217604] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.217630] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.217632] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.217659] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.217663] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.217697] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.217703] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.217700] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.217720] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.217764] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.217767] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.217770] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.217786] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.217914] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.217918] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.217929] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.217933] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.217958] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.217961] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.217961] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.217993] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.218026] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.218028] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.218031] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.218042] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.218181] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.218184] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.218195] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.218200] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.218224] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.218228] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.546848] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.546848] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.591765] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.591766] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.725672] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.726510] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.770714] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.770717] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.770779] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.770807] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.770867] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.770870] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.770884] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.770894] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.770950] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.770954] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.771989] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.772203] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.772247] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.772250] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.772273] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.787313] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.787358] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.787362] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.787384] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.787644] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.787704] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.787707] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.787721] [c318-001:1866939:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14b095201080
[1719264277.787726] [c318-001:1866940:0]    tl_nccl_team.c:293  TL_NCCL DEBUG init coll task 0x14f5a5201080
[1719264277.787796] [c318-001:1866940:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14f5a5201080
[1719264277.787799] [c318-001:1866939:0]    tl_nccl_coll.c:219  TL_NCCL DEBUG finalizing coll task 0x14b095201080
[1719264277.787851] [c318-001:1866940:0] ucc_coll_score_map.c:144  UCC  DEBUG coll Bcast is not supported for TL_SHM, fallback TL_UCP
[1719264277.787893] [c318-001:1866939:0] ucc_coll_score_map.c:144  UCC  DEBUG coll Bcast is not supported for TL_SHM, fallback TL_UCP
print_host_matrix : Output matrix X
0.03
0.04
0.14
0.07
0.03
0.08
0.09
0.20
0.19
0.42

|b - A*x|_inf = 2.220446E-16
|x|_inf = 1.273718E+00
|b|_inf = 1.000000E+01
|A|_inf = 1.674133E+01
|b - A*x|/(|A|*|x|+|b|) = 7.088702E-18

[1719264277.788895] [c318-001:1866940:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0xe62990
[1719264277.788903] [c318-001:1866940:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x3ca6b10
[1719264277.788912] [c318-001:1866940:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x17ae010
[1719264277.788975] [c318-001:1866939:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0x2655990
[1719264277.788982] [c318-001:1866939:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x544f840
[1719264277.788988] [c318-001:1866939:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x3689cc0
[1719264277.788992] [c318-001:1866939:0]        ucc_team.c:528  UCC  INFO  team destroyed, team_id 32769
[1719264277.812779] [c318-001:1866939:0]    tl_nccl_team.c:65   TL_NCCL DEBUG finalizing tl team: 0x26524d0
[1719264277.812799] [c318-001:1866939:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0x2655760
[1719264277.812805] [c318-001:1866939:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x54ad360
[1719264277.812812] [c318-001:1866939:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x2e429b0
[1719264277.812817] [c318-001:1866939:0]        ucc_team.c:528  UCC  INFO  team destroyed, team_id 32770
[1719264277.812833] [c318-001:1866939:0]    tl_self_team.c:25   TL_SELF DEBUG finalizing tl team: 0x3f6b610
[1719264277.812837] [c318-001:1866939:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x2627e50
[1719264277.812839] [c318-001:1866939:0]        ucc_team.c:528  UCC  INFO  team destroyed, team_id 32771
[1719264277.835520] [c318-001:1866939:0]    tl_nccl_team.c:65   TL_NCCL DEBUG finalizing tl team: 0x54b6c30
[1719264277.835537] [c318-001:1866939:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0x2655530
[1719264277.835543] [c318-001:1866939:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x54b7b70
[1719264277.835549] [c318-001:1866939:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x213b190
[1719264277.835552] [c318-001:1866939:0]        ucc_team.c:528  UCC  INFO  team destroyed, team_id 32772
[1719264277.835562] [c318-001:1866939:0]    tl_self_team.c:25   TL_SELF DEBUG finalizing tl team: 0x3f6b4a0
[1719264277.835566] [c318-001:1866939:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x54b89a0
[1719264277.835569] [c318-001:1866939:0]        ucc_team.c:528  UCC  INFO  team destroyed, team_id 32773
[1719264277.835581] [c318-001:1866939:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0x2655250
[1719264277.835584] [c318-001:1866939:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x54c1700
[1719264277.835587] [c318-001:1866939:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x54c1260
[1719264277.835591] [c318-001:1866939:0]        ucc_team.c:528  UCC  INFO  team destroyed, team_id 32774
[1719264278.020109] [c318-001:1866940:0]    tl_nccl_team.c:65   TL_NCCL DEBUG finalizing tl team: 0xe5f4d0
[1719264278.020144] [c318-001:1866940:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0xe62760
[1719264278.020153] [c318-001:1866940:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x3c66850
[1719264278.020162] [c318-001:1866940:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x164ec60
[1719264278.020182] [c318-001:1866940:0]    tl_self_team.c:25   TL_SELF DEBUG finalizing tl team: 0x277d150
[1719264278.020187] [c318-001:1866940:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0xe34e50
[1719264278.020189] [c318-001:1866940:0]        ucc_team.c:528  UCC  INFO  team destroyed, team_id 32771
[1719264278.024440] [c318-001:1866940:0]    tl_nccl_team.c:65   TL_NCCL DEBUG finalizing tl team: 0x3cc8010
[1719264278.024459] [c318-001:1866940:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0xe62530
[1719264278.024464] [c318-001:1866940:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x3cc8f50
[1719264278.024469] [c318-001:1866940:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x948190
[1719264278.024479] [c318-001:1866940:0]    tl_self_team.c:25   TL_SELF DEBUG finalizing tl team: 0x1352780
[1719264278.024483] [c318-001:1866940:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x3cc9aa0
[1719264278.024486] [c318-001:1866940:0]        ucc_team.c:528  UCC  INFO  team destroyed, team_id 32773
[1719264278.024502] [c318-001:1866940:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0xe62250
[1719264278.024507] [c318-001:1866940:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x3cd27b0
[1719264278.024513] [c318-001:1866940:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x3cd2310
[1719264278.031646] [c318-001:1866940:0]    tl_nccl_team.c:65   TL_NCCL DEBUG finalizing tl team: 0x1e99e10
[1719264278.031662] [c318-001:1866940:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0x277a490
[1719264278.031666] [c318-001:1866940:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x34ff090
[1719264278.031672] [c318-001:1866940:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x1f4fde0
[1719264278.031683] [c318-001:1866940:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x340e050
[1719264278.031688] [c318-001:1866940:0] cl_basic_context.c:57   CL_BASIC DEBUG finalizing cl context: 0x29e7c10
[1719264278.031694] [c318-001:1866940:0] tl_self_context.c:39   TL_SELF DEBUG finalizing tl context: 0x2d76ab0
[1719264278.031710] [c318-001:1866940:0]  tl_shm_context.c:39   TL_SHM DEBUG finalizing tl context: 0x2d76bc0
[1719264278.031715] [c318-001:1866940:0]  tl_ucp_context.c:380  TL_UCP DEBUG finalizing tl context: 0x29c8010
[1719264278.032496] [c318-001:1866939:0]    tl_nccl_team.c:65   TL_NCCL DEBUG finalizing tl team: 0x373fad0
[1719264278.032524] [c318-001:1866939:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0x3f69730
[1719264278.032531] [c318-001:1866939:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x4cea090
[1719264278.032536] [c318-001:1866939:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x373fc20
[1719264278.032539] [c318-001:1866939:0]        ucc_team.c:528  UCC  INFO  team destroyed, team_id 32768
[1719264278.032552] [c318-001:1866939:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x4b62ac0
[1719264278.032556] [c318-001:1866939:0] cl_basic_context.c:57   CL_BASIC DEBUG finalizing cl context: 0x41d6fb0
[1719264278.032563] [c318-001:1866939:0] tl_self_context.c:39   TL_SELF DEBUG finalizing tl context: 0x456dea0
[1719264278.032573] [c318-001:1866939:0]  tl_shm_context.c:39   TL_SHM DEBUG finalizing tl context: 0x4376510
[1719264278.032578] [c318-001:1866939:0]  tl_ucp_context.c:380  TL_UCP DEBUG finalizing tl context: 0x41b7010
[1719264278.051167] [c318-001:1866939:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x4376060
[1719264278.051177] [c318-001:1866939:0] cl_basic_context.c:57   CL_BASIC DEBUG finalizing cl context: 0x3f68830
[1719264278.051183] [c318-001:1866939:0] tl_cuda_context.c:81   TL_CUDA DEBUG finalizing tl context: 0x39429d0
[1719264278.051192] [c318-001:1866939:0] tl_nccl_context.c:211  TL_NCCL DEBUG finalizing tl context: 0x3736120
[1719264278.051201] [c318-001:1866940:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x2d76470
[1719264278.051210] [c318-001:1866940:0] cl_basic_context.c:57   CL_BASIC DEBUG finalizing cl context: 0x2779590
[1719264278.051217] [c318-001:1866940:0] tl_cuda_context.c:81   TL_CUDA DEBUG finalizing tl context: 0x2152ef0
[1719264278.051223] [c318-001:1866940:0] tl_nccl_context.c:211  TL_NCCL DEBUG finalizing tl context: 0x1f46270
[1719264278.051566] [c318-001:1866939:0] tl_self_context.c:39   TL_SELF DEBUG finalizing tl context: 0x3f5e8d0
[1719264278.051572] [c318-001:1866939:0]  tl_shm_context.c:39   TL_SHM DEBUG finalizing tl context: 0x3f5ea00
[1719264278.051577] [c318-001:1866939:0]  tl_ucp_context.c:380  TL_UCP DEBUG finalizing tl context: 0x3712910
[1719264278.051697] [c318-001:1866940:0] tl_self_context.c:39   TL_SELF DEBUG finalizing tl context: 0x276f560
[1719264278.051703] [c318-001:1866940:0]  tl_shm_context.c:39   TL_SHM DEBUG finalizing tl context: 0x276f690
[1719264278.051707] [c318-001:1866940:0]  tl_ucp_context.c:380  TL_UCP DEBUG finalizing tl context: 0x1f22a60
[1719264278.068473] [c318-001:1866939:0]     tl_cuda_lib.c:41   TL_CUDA DEBUG finalizing lib object: 0x3c9eed0
[1719264278.068480] [c318-001:1866939:0]     tl_nccl_lib.c:22   TL_NCCL DEBUG finalizing lib object: 0x3f5c3f0
[1719264278.068484] [c318-001:1866939:0]     tl_self_lib.c:26   TL_SELF DEBUG finalizing lib object: 0x3c89030
[1719264278.068488] [c318-001:1866939:0]      tl_shm_lib.c:30   TL_SHM DEBUG finalizing lib object: 0x3f28bb0
[1719264278.068493] [c318-001:1866939:0]      tl_ucp_lib.c:83   TL_UCP DEBUG finalizing lib object: 0x3f28e70
[1719264278.068497] [c318-001:1866939:0]    cl_basic_lib.c:26   CL_BASIC DEBUG finalizing lib object: 0x37479f0
[1719264278.068508] [c318-001:1866940:0]     tl_cuda_lib.c:41   TL_CUDA DEBUG finalizing lib object: 0x24afb40
[1719264278.068515] [c318-001:1866940:0]     tl_nccl_lib.c:22   TL_NCCL DEBUG finalizing lib object: 0x2739f40
[1719264278.068519] [c318-001:1866940:0]     tl_self_lib.c:26   TL_SELF DEBUG finalizing lib object: 0x24999d0
[1719264278.068523] [c318-001:1866940:0]      tl_shm_lib.c:30   TL_SHM DEBUG finalizing lib object: 0x2739b40
[1719264278.068527] [c318-001:1866940:0]      tl_ucp_lib.c:83   TL_UCP DEBUG finalizing lib object: 0x27661c0
[1719264278.068530] [c318-001:1866940:0]    cl_basic_lib.c:26   CL_BASIC DEBUG finalizing lib object: 0x1f57f10
[1719264278.069623] [c318-001:1866939:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0x2655be0
[1719264278.069630] [c318-001:1866939:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x32b1de0
[1719264278.069636] [c318-001:1866939:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x3203d60
[1719264278.069640] [c318-001:1866939:0]        ucc_team.c:528  UCC  INFO  team destroyed, team_id 32768
[1719264278.069647] [c318-001:1866939:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x2e107b0
[1719264278.069651] [c318-001:1866939:0] cl_basic_context.c:57   CL_BASIC DEBUG finalizing cl context: 0x2b512c0
[1719264278.069655] [c318-001:1866939:0] cl_hier_context.c:75   CL_HIER DEBUG finalizing cl context: 0x2b4aad0
[1719264278.069664] [c318-001:1866939:0] tl_self_context.c:39   TL_SELF DEBUG finalizing tl context: 0x2b47350
[1719264278.069667] [c318-001:1866939:0]  tl_shm_context.c:39   TL_SHM DEBUG finalizing tl context: 0x2b47440
[1719264278.069671] [c318-001:1866939:0]  tl_ucp_context.c:380  TL_UCP DEBUG finalizing tl context: 0x2b0ebb0
[1719264278.069773] [c318-001:1866940:0]     tl_shm_team.c:534  TL_SHM DEBUG finalizing tl team: 0xe62be0
[1719264278.069780] [c318-001:1866940:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x1ad0fc0
[1719264278.069787] [c318-001:1866940:0]   cl_basic_team.c:61   CL_BASIC DEBUG finalizing cl team: 0x16c91f0
[1719264278.069796] [c318-001:1866940:0]     tl_ucp_team.c:110  TL_UCP DEBUG finalizing tl team: 0x161c7a0
[1719264278.069800] [c318-001:1866940:0] cl_basic_context.c:57   CL_BASIC DEBUG finalizing cl context: 0x1354f80
[1719264278.069804] [c318-001:1866940:0] cl_hier_context.c:75   CL_HIER DEBUG finalizing cl context: 0x1349230
[1719264278.069810] [c318-001:1866940:0] tl_self_context.c:39   TL_SELF DEBUG finalizing tl context: 0x1345bc0
[1719264278.069813] [c318-001:1866940:0]  tl_shm_context.c:39   TL_SHM DEBUG finalizing tl context: 0x1345cb0
[1719264278.069817] [c318-001:1866940:0]  tl_ucp_context.c:380  TL_UCP DEBUG finalizing tl context: 0x12f9850
[1719264278.086467] [c318-001:1866939:0]     tl_cuda_lib.c:41   TL_CUDA DEBUG finalizing lib object: 0x2ad4c30
[1719264278.086476] [c318-001:1866939:0]     tl_self_lib.c:26   TL_SELF DEBUG finalizing lib object: 0x2b20130
[1719264278.086479] [c318-001:1866939:0]      tl_shm_lib.c:30   TL_SHM DEBUG finalizing lib object: 0x2b46e20
[1719264278.086482] [c318-001:1866939:0]      tl_ucp_lib.c:83   TL_UCP DEBUG finalizing lib object: 0x2516e20
[1719264278.086486] [c318-001:1866939:0]    cl_basic_lib.c:26   CL_BASIC DEBUG finalizing lib object: 0x2b46040
[1719264278.086491] [c318-001:1866939:0]     cl_hier_lib.c:61   CL_HIER DEBUG finalizing lib object: 0x27d5c20
[1719264278.086565] [c318-001:1866940:0]     tl_cuda_lib.c:41   TL_CUDA DEBUG finalizing lib object: 0x12f9450
[1719264278.086572] [c318-001:1866940:0]     tl_self_lib.c:26   TL_SELF DEBUG finalizing lib object: 0x12fbd90
[1719264278.086576] [c318-001:1866940:0]      tl_shm_lib.c:30   TL_SHM DEBUG finalizing lib object: 0x1345690
[1719264278.086579] [c318-001:1866940:0]      tl_ucp_lib.c:83   TL_UCP DEBUG finalizing lib object: 0xd23e20
[1719264278.086582] [c318-001:1866940:0]    cl_basic_lib.c:26   CL_BASIC DEBUG finalizing lib object: 0x13448b0
[1719264278.086585] [c318-001:1866940:0]     cl_hier_lib.c:61   CL_HIER DEBUG finalizing lib object: 0xfe2ac0

I think it is the unsupported backend thing that you mentioned.

mrogowski commented 2 months ago

I do not see the RDMA error here. So if you run with UCC_TLS=^mlx5,sharp but without UCC_LOG_LEVEL=DEBUG, the output looks clean?

s769 commented 2 months ago

Yes, the output is clean (aside from those warnings about "initialized with different thread mode"). Thanks for your help.