Reduce/Allreduce issues on cyt_rdma

I'm getting errors on repeated runs of Reduce and Allreduce:

...
Pass accl barrier
host measured durationUs:91.724
2th item is incorrect! (1.000000 != 2.000000)
3th item is incorrect! (2.000000 != 4.000000)
4th item is incorrect! (3.000000 != 6.000000)
5th item is incorrect! (4.000000 != 8.000000)
...

The first run succeeds.

I'm using a sligthly modified version of the script https://github.com/Xilinx/ACCL/blob/dev/test/host/Coyote/run_scripts/run.sh on commit https://github.com/Xilinx/ACCL/commit/a0ba7ea3040b026359d5f790acb7f83b67e29645

Output of first run(allreduce):

stdout

stderr

``` XRT build version: 2.13.466 Build hash: f5505e402c2ca1ffe45eb6d3a9399b23a0dc8776 Build date: 2022-04-14 17:43:11 Git branch: 2022.1 PID: 21032 UID: 500207 [Tue May 14 12:09:44 2024 GMT] HOST: alveo-u55c-07.inf.ethz.ch EXE: /pub/scratch/lawirz/XACCL/integrations/pytorch_ddp/accl/test/host/Coyote/accl_on_coyote [XRT] ERROR: No devices found [XRT] ERROR: No devices found [XRT] ERROR: No devices found ACLL DEBUG: aquiring cProc: targetRegion: 0, cPid: 0 ACLL DEBUG: aquiring qProc: targetRegion: 0, cPid: 1 ACLL DEBUG: aquiring qProc: targetRegion: 0, cPid: 2 CCLO HWID: 2696576574 at 0x0 CCLO source commit (first 24b): a0ba7e CCLO Capabilities: Stack type: RDMA Internal DMA:True External DMA:False Reduction:True Compression:True Kernel Streams:True Debug:False Doing a soft reset Configuring Eager RX Buffers get_device_type: coyote_device get_device_type: coyote_device CoyoteBuffer contructor called! page_size:2097152, buffer_size:64,n_pages:1 Allocation successful! Allocated buffer: 7fe622c00000, Size: 64 calling offload: 7fe622c00000, size: 64 get_device_type: coyote_device get_device_type: coyote_device CoyoteBuffer contructor called! page_size:2097152, buffer_size:64,n_pages:1 Allocation successful! Allocated buffer: 7fe622a00000, Size: 64 calling offload: 7fe622a00000, size: 64 Configuring Rendezvous Spare Buffers get_device_type: coyote_device get_device_type: coyote_device CoyoteBuffer contructor called! page_size:2097152, buffer_size:4194304,n_pages:2 Allocation successful! Allocated buffer: 7fe622600000, Size: 4194304 calling offload: 7fe622600000, size: 4194304 get_device_type: coyote_device get_device_type: coyote_device CoyoteBuffer contructor called! page_size:2097152, buffer_size:4194304,n_pages:2 Allocation successful! Allocated buffer: 7fe622200000, Size: 4194304 calling offload: 7fe622200000, size: 4194304 get_device_type: coyote_device get_device_type: coyote_device CoyoteBuffer contructor called! page_size:2097152, buffer_size:4194304,n_pages:2 Allocation successful! Allocated buffer: 7fe621e00000, Size: 4194304 calling offload: 7fe621e00000, size: 4194304 Configuring a communicator Configuring arithmetic Configuring collective tuning parameters CCLO configured Set timeout Set max eager size: 64 Set max rendezvous reduce size: 4194304 Accelerator ready! Communicator 0 (0x40): local rank: 0 number of ranks: 2 > rank 0 (ip 10.253.74.92:5005 ; session 0 ; max segment size 4194304) : <- inbound seq number 0, -> outbound seq number 0 > rank 1 (ip 10.253.74.96:5005 ; session 2 ; max segment size 4194304) : <- inbound seq number 0, -> outbound seq number 0 Rank 0 passed last barrier before test! CCLO address: 0 rx address: 4 Spare RX Buffer 0: address: 0x7fe622c00000 status: ENQUEUED occupancy: 0/64 MPI tag: 0 seq: 0 src: 0 Spare RX Buffer 1: address: 0x7fe622a00000 status: ENQUEUED occupancy: 0/64 MPI tag: 0 seq: 0 src: 0 CoyoteBuffer contructor called! page_size:2097152, buffer_size:2048,n_pages:1 Allocation successful! Allocated buffer: 7fe621c00000, Size: 2048 CoyoteBuffer contructor called! page_size:2097152, buffer_size:2048,n_pages:1 Allocation successful! Allocated buffer: 7fe621a00000, Size: 2048 Reducing data... Free user buffer from cProc cPid:0, buffer_size:2048,7fe621c00000 Free user buffer from cProc cPid:0, buffer_size:2048,7fe621a00000 Communicator 0 (0x40): local rank: 0 number of ranks: 2 > rank 0 (ip 10.253.74.92:5005 ; session 0 ; max segment size 4194304) : <- inbound seq number 0, -> outbound seq number 0 > rank 1 (ip 10.253.74.96:5005 ; session 2 ; max segment size 4194304) : <- inbound seq number 64, -> outbound seq number 64 CCLO address: 0 rx address: 4 Spare RX Buffer 0: address: 0x7fe622c00000 status: ENQUEUED occupancy: 32/64 MPI tag: ffffffff seq: 62 src: 1 Spare RX Buffer 1: address: 0x7fe622a00000 status: ENQUEUED occupancy: 32/64 MPI tag: ffffffff seq: 63 src: 1 Removing CCLO object at 0 Doing a soft reset Free user buffer from cProc cPid:0, buffer_size:64,7fe622c00000 Free user buffer from cProc cPid:0, buffer_size:64,7fe622a00000 Free user buffer from cProc cPid:0, buffer_size:4194304,7fe622600000 Free user buffer from cProc cPid:0, buffer_size:4194304,7fe622200000 Free user buffer from cProc cPid:0, buffer_size:4194304,7fe621e00000 ```

Second run:

stdout

``` Arguments: '../accl_on_coyote' '-d' '-f' '-r' '-z' '1' '-y' '10' '-c' '512' '-l' './accl_log/fpga' '-p' '1' '-n' '1' Running ACCL test in coyote... Initializing MPI... Reading MPI rank and size values... Parsing options Hardware rdma mode count:512 rxbuf_size:4194304 seg_size:4194304 num_rxbufmem:2 Getting MPI Processor name... [process 0] rank 0 size 2 alveo-u55c-07.inf.ethz.ch Testing ACCL base functionality... 10.253.74.92 10.253.74.96 Initializing QP connections... Exchanging QP... Local rank 0 sending local QP to remote rank 1 Local rank 0 receiving remote QP from remote rank 1 Queue Pair: id: 1 Local Queue: local: QPN 0x000001, PSN 0x709eba, VADDR 00007fe2c6c00000, SIZE 00200000, IP 0x0afd4a5c, Remote Queue: remote: QPN 0x000002, PSN 0xd26325, VADDR 00007f9402e00000, SIZE 00200000, IP 0x0afd4a60, rank: 0 FPGA IP: afd4a5c Rendezvous Protocol sw nop time [us]:87.215 hw nop time [ns]:940 Start allreduce test and reduce function 0... Repetition 0 Pass accl barrier host measured durationUs:79.17 2th item is incorrect! (1.000000 != 2.000000) 3th item is incorrect! (2.000000 != 4.000000) 4th item is incorrect! (3.000000 != 6.000000) 5th item is incorrect! (4.000000 != 8.000000) 6th item is incorrect! (5.000000 != 10.000000) 7th item is incorrect! (6.000000 != 12.000000) 8th item is incorrect! (7.000000 != 14.000000) 9th item is incorrect! (8.000000 != 16.000000) 10th item is incorrect! (9.000000 != 18.000000) 11th item is incorrect! (10.000000 != 20.000000) 12th item is incorrect! (11.000000 != 22.000000) 13th item is incorrect! (12.000000 != 24.000000) 14th item is incorrect! (13.000000 != 26.000000) 15th item is incorrect! (14.000000 != 28.000000) 16th item is incorrect! (15.000000 != 30.000000) 17th item is incorrect! (16.000000 != 32.000000) 18th item is incorrect! (17.000000 != 34.000000) 19th item is incorrect! (18.000000 != 36.000000) 20th item is incorrect! (19.000000 != 38.000000) 21th item is incorrect! (20.000000 != 40.000000) 22th item is incorrect! (21.000000 != 42.000000) 23th item is incorrect! (22.000000 != 44.000000) 24th item is incorrect! (23.000000 != 46.000000) 25th item is incorrect! (24.000000 != 48.000000) 26th item is incorrect! (25.000000 != 50.000000) 27th item is incorrect! (26.000000 != 52.000000) 28th item is incorrect! (27.000000 != 54.000000) 29th item is incorrect! (28.000000 != 56.000000) 30th item is incorrect! (29.000000 != 58.000000) 31th item is incorrect! (30.000000 != 60.000000) 32th item is incorrect! (31.000000 != 62.000000) 33th item is incorrect! (32.000000 != 64.000000) 34th item is incorrect! (33.000000 != 66.000000) 35th item is incorrect! (34.000000 != 68.000000) 36th item is incorrect! (35.000000 != 70.000000) 37th item is incorrect! (36.000000 != 72.000000) 38th item is incorrect! (37.000000 != 74.000000) 39th item is incorrect! (38.000000 != 76.000000) 40th item is incorrect! (39.000000 != 78.000000) 41th item is incorrect! (40.000000 != 80.000000) 42th item is incorrect! (41.000000 != 82.000000) 43th item is incorrect! (42.000000 != 84.000000) 44th item is incorrect! (43.000000 != 86.000000) 45th item is incorrect! (44.000000 != 88.000000) 46th item is incorrect! (45.000000 != 90.000000) 47th item is incorrect! (46.000000 != 92.000000) 48th item is incorrect! (47.000000 != 94.000000) 49th item is incorrect! (48.000000 != 96.000000) 50th item is incorrect! (49.000000 != 98.000000) 51th item is incorrect! (50.000000 != 100.000000) 52th item is incorrect! (51.000000 != 102.000000) 53th item is incorrect! (52.000000 != 104.000000) 54th item is incorrect! (53.000000 != 106.000000) 55th item is incorrect! (54.000000 != 108.000000) 56th item is incorrect! (55.000000 != 110.000000) 57th item is incorrect! (56.000000 != 112.000000) 58th item is incorrect! (57.000000 != 114.000000) 59th item is incorrect! (58.000000 != 116.000000) 60th item is incorrect! (59.000000 != 118.000000) 61th item is incorrect! (60.000000 != 120.000000) 62th item is incorrect! (61.000000 != 122.000000) 63th item is incorrect! (62.000000 != 124.000000) 64th item is incorrect! (63.000000 != 126.000000) 65th item is incorrect! (64.000000 != 128.000000) 66th item is incorrect! (65.000000 != 130.000000) 67th item is incorrect! (66.000000 != 132.000000) 68th item is incorrect! (67.000000 != 134.000000) 69th item is incorrect! (68.000000 != 136.000000) 70th item is incorrect! (69.000000 != 138.000000) 71th item is incorrect! (70.000000 != 140.000000) 72th item is incorrect! (71.000000 != 142.000000) 73th item is incorrect! (72.000000 != 144.000000) 74th item is incorrect! (73.000000 != 146.000000) 75th item is incorrect! (74.000000 != 148.000000) 76th item is incorrect! (75.000000 != 150.000000) 77th item is incorrect! (76.000000 != 152.000000) 78th item is incorrect! (77.000000 != 154.000000) 79th item is incorrect! (78.000000 != 156.000000) 80th item is incorrect! (79.000000 != 158.000000) 81th item is incorrect! (80.000000 != 160.000000) 82th item is incorrect! (81.000000 != 162.000000) 83th item is incorrect! (82.000000 != 164.000000) 84th item is incorrect! (83.000000 != 166.000000) 85th item is incorrect! (84.000000 != 168.000000) 86th item is incorrect! (85.000000 != 170.000000) 87th item is incorrect! (86.000000 != 172.000000) 88th item is incorrect! (87.000000 != 174.000000) 89th item is incorrect! (88.000000 != 176.000000) 90th item is incorrect! (89.000000 != 178.000000) 91th item is incorrect! (90.000000 != 180.000000) 92th item is incorrect! (91.000000 != 182.000000) 93th item is incorrect! (92.000000 != 184.000000) 94th item is incorrect! (93.000000 != 186.000000) 95th item is incorrect! (94.000000 != 188.000000) 96th item is incorrect! (95.000000 != 190.000000) 97th item is incorrect! (96.000000 != 192.000000) 98th item is incorrect! (97.000000 != 194.000000) 99th item is incorrect! (98.000000 != 196.000000) 100th item is incorrect! (99.000000 != 198.000000) 101th item is incorrect! (100.000000 != 200.000000) 102th item is incorrect! (101.000000 != 202.000000) 103th item is incorrect! (102.000000 != 204.000000) 104th item is incorrect! (103.000000 != 206.000000) 105th item is incorrect! (104.000000 != 208.000000) 106th item is incorrect! (105.000000 != 210.000000) 107th item is incorrect! (106.000000 != 212.000000) 108th item is incorrect! (107.000000 != 214.000000) 109th item is incorrect! (108.000000 != 216.000000) 110th item is incorrect! (109.000000 != 218.000000) 111th item is incorrect! (110.000000 != 220.000000) 112th item is incorrect! (111.000000 != 222.000000) 113th item is incorrect! (112.000000 != 224.000000) 114th item is incorrect! (113.000000 != 226.000000) 115th item is incorrect! (114.000000 != 228.000000) 116th item is incorrect! (115.000000 != 230.000000) 117th item is incorrect! (116.000000 != 232.000000) 118th item is incorrect! (117.000000 != 234.000000) 119th item is incorrect! (118.000000 != 236.000000) 120th item is incorrect! (119.000000 != 238.000000) 121th item is incorrect! (120.000000 != 240.000000) 122th item is incorrect! (121.000000 != 242.000000) 123th item is incorrect! (122.000000 != 244.000000) 124th item is incorrect! (123.000000 != 246.000000) 125th item is incorrect! (124.000000 != 248.000000) 126th item is incorrect! (125.000000 != 250.000000) 127th item is incorrect! (126.000000 != 252.000000) 128th item is incorrect! (127.000000 != 254.000000) 129th item is incorrect! (128.000000 != 256.000000) 130th item is incorrect! (129.000000 != 258.000000) 131th item is incorrect! (130.000000 != 260.000000) 132th item is incorrect! (131.000000 != 262.000000) 133th item is incorrect! (132.000000 != 264.000000) 134th item is incorrect! (133.000000 != 266.000000) 135th item is incorrect! (134.000000 != 268.000000) 136th item is incorrect! (135.000000 != 270.000000) 137th item is incorrect! (136.000000 != 272.000000) 138th item is incorrect! (137.000000 != 274.000000) 139th item is incorrect! (138.000000 != 276.000000) 140th item is incorrect! (139.000000 != 278.000000) 141th item is incorrect! (140.000000 != 280.000000) 142th item is incorrect! (141.000000 != 282.000000) 143th item is incorrect! (142.000000 != 284.000000) 144th item is incorrect! (143.000000 != 286.000000) 145th item is incorrect! (144.000000 != 288.000000) 146th item is incorrect! (145.000000 != 290.000000) 147th item is incorrect! (146.000000 != 292.000000) 148th item is incorrect! (147.000000 != 294.000000) 149th item is incorrect! (148.000000 != 296.000000) 150th item is incorrect! (149.000000 != 298.000000) 151th item is incorrect! (150.000000 != 300.000000) 152th item is incorrect! (151.000000 != 302.000000) 153th item is incorrect! (152.000000 != 304.000000) 154th item is incorrect! (153.000000 != 306.000000) 155th item is incorrect! (154.000000 != 308.000000) 156th item is incorrect! (155.000000 != 310.000000) 157th item is incorrect! (156.000000 != 312.000000) 158th item is incorrect! (157.000000 != 314.000000) 159th item is incorrect! (158.000000 != 316.000000) 160th item is incorrect! (159.000000 != 318.000000) 161th item is incorrect! (160.000000 != 320.000000) 162th item is incorrect! (161.000000 != 322.000000) 163th item is incorrect! (162.000000 != 324.000000) 164th item is incorrect! (163.000000 != 326.000000) 165th item is incorrect! (164.000000 != 328.000000) 166th item is incorrect! (165.000000 != 330.000000) 167th item is incorrect! (166.000000 != 332.000000) 168th item is incorrect! (167.000000 != 334.000000) 169th item is incorrect! (168.000000 != 336.000000) 170th item is incorrect! (169.000000 != 338.000000) 171th item is incorrect! (170.000000 != 340.000000) 172th item is incorrect! (171.000000 != 342.000000) 173th item is incorrect! (172.000000 != 344.000000) 174th item is incorrect! (173.000000 != 346.000000) 175th item is incorrect! (174.000000 != 348.000000) 176th item is incorrect! (175.000000 != 350.000000) 177th item is incorrect! (176.000000 != 352.000000) 178th item is incorrect! (177.000000 != 354.000000) 179th item is incorrect! (178.000000 != 356.000000) 180th item is incorrect! (179.000000 != 358.000000) 181th item is incorrect! (180.000000 != 360.000000) 182th item is incorrect! (181.000000 != 362.000000) 183th item is incorrect! (182.000000 != 364.000000) 184th item is incorrect! (183.000000 != 366.000000) 185th item is incorrect! (184.000000 != 368.000000) 186th item is incorrect! (185.000000 != 370.000000) 187th item is incorrect! (186.000000 != 372.000000) 188th item is incorrect! (187.000000 != 374.000000) 189th item is incorrect! (188.000000 != 376.000000) 190th item is incorrect! (189.000000 != 378.000000) 191th item is incorrect! (190.000000 != 380.000000) 192th item is incorrect! (191.000000 != 382.000000) 193th item is incorrect! (192.000000 != 384.000000) 194th item is incorrect! (193.000000 != 386.000000) 195th item is incorrect! (194.000000 != 388.000000) 196th item is incorrect! (195.000000 != 390.000000) 197th item is incorrect! (196.000000 != 392.000000) 198th item is incorrect! (197.000000 != 394.000000) 199th item is incorrect! (198.000000 != 396.000000) 200th item is incorrect! (199.000000 != 398.000000) 201th item is incorrect! (200.000000 != 400.000000) 202th item is incorrect! (201.000000 != 402.000000) 203th item is incorrect! (202.000000 != 404.000000) 204th item is incorrect! (203.000000 != 406.000000) 205th item is incorrect! (204.000000 != 408.000000) 206th item is incorrect! (205.000000 != 410.000000) 207th item is incorrect! (206.000000 != 412.000000) 208th item is incorrect! (207.000000 != 414.000000) 209th item is incorrect! (208.000000 != 416.000000) 210th item is incorrect! (209.000000 != 418.000000) 211th item is incorrect! (210.000000 != 420.000000) 212th item is incorrect! (211.000000 != 422.000000) 213th item is incorrect! (212.000000 != 424.000000) 214th item is incorrect! (213.000000 != 426.000000) 215th item is incorrect! (214.000000 != 428.000000) 216th item is incorrect! (215.000000 != 430.000000) 217th item is incorrect! (216.000000 != 432.000000) 218th item is incorrect! (217.000000 != 434.000000) 219th item is incorrect! (218.000000 != 436.000000) 220th item is incorrect! (219.000000 != 438.000000) 221th item is incorrect! (220.000000 != 440.000000) 222th item is incorrect! (221.000000 != 442.000000) 223th item is incorrect! (222.000000 != 444.000000) 224th item is incorrect! (223.000000 != 446.000000) 225th item is incorrect! (224.000000 != 448.000000) 226th item is incorrect! (225.000000 != 450.000000) 227th item is incorrect! (226.000000 != 452.000000) 228th item is incorrect! (227.000000 != 454.000000) 229th item is incorrect! (228.000000 != 456.000000) 230th item is incorrect! (229.000000 != 458.000000) 231th item is incorrect! (230.000000 != 460.000000) 232th item is incorrect! (231.000000 != 462.000000) 233th item is incorrect! (232.000000 != 464.000000) 234th item is incorrect! (233.000000 != 466.000000) 235th item is incorrect! (234.000000 != 468.000000) 236th item is incorrect! (235.000000 != 470.000000) 237th item is incorrect! (236.000000 != 472.000000) 238th item is incorrect! (237.000000 != 474.000000) 239th item is incorrect! (238.000000 != 476.000000) 240th item is incorrect! (239.000000 != 478.000000) 241th item is incorrect! (240.000000 != 480.000000) 242th item is incorrect! (241.000000 != 482.000000) 243th item is incorrect! (242.000000 != 484.000000) 244th item is incorrect! (243.000000 != 486.000000) 245th item is incorrect! (244.000000 != 488.000000) 246th item is incorrect! (245.000000 != 490.000000) 247th item is incorrect! (246.000000 != 492.000000) 248th item is incorrect! (247.000000 != 494.000000) 249th item is incorrect! (248.000000 != 496.000000) 250th item is incorrect! (249.000000 != 498.000000) 251th item is incorrect! (250.000000 != 500.000000) 252th item is incorrect! (251.000000 != 502.000000) 253th item is incorrect! (252.000000 != 504.000000) 254th item is incorrect! (253.000000 != 506.000000) 255th item is incorrect! (254.000000 != 508.000000) 256th item is incorrect! (255.000000 != 510.000000) 257th item is incorrect! (256.000000 != 512.000000) 258th item is incorrect! (257.000000 != 514.000000) 259th item is incorrect! (258.000000 != 516.000000) 260th item is incorrect! (259.000000 != 518.000000) 261th item is incorrect! (260.000000 != 520.000000) 262th item is incorrect! (261.000000 != 522.000000) 263th item is incorrect! (262.000000 != 524.000000) 264th item is incorrect! (263.000000 != 526.000000) 265th item is incorrect! (264.000000 != 528.000000) 266th item is incorrect! (265.000000 != 530.000000) 267th item is incorrect! (266.000000 != 532.000000) 268th item is incorrect! (267.000000 != 534.000000) 269th item is incorrect! (268.000000 != 536.000000) 270th item is incorrect! (269.000000 != 538.000000) 271th item is incorrect! (270.000000 != 540.000000) 272th item is incorrect! (271.000000 != 542.000000) 273th item is incorrect! (272.000000 != 544.000000) 274th item is incorrect! (273.000000 != 546.000000) 275th item is incorrect! (274.000000 != 548.000000) 276th item is incorrect! (275.000000 != 550.000000) 277th item is incorrect! (276.000000 != 552.000000) 278th item is incorrect! (277.000000 != 554.000000) 279th item is incorrect! (278.000000 != 556.000000) 280th item is incorrect! (279.000000 != 558.000000) 281th item is incorrect! (280.000000 != 560.000000) 282th item is incorrect! (281.000000 != 562.000000) 283th item is incorrect! (282.000000 != 564.000000) 284th item is incorrect! (283.000000 != 566.000000) 285th item is incorrect! (284.000000 != 568.000000) 286th item is incorrect! (285.000000 != 570.000000) 287th item is incorrect! (286.000000 != 572.000000) 288th item is incorrect! (287.000000 != 574.000000) 289th item is incorrect! (288.000000 != 576.000000) 290th item is incorrect! (289.000000 != 578.000000) 291th item is incorrect! (290.000000 != 580.000000) 292th item is incorrect! (291.000000 != 582.000000) 293th item is incorrect! (292.000000 != 584.000000) 294th item is incorrect! (293.000000 != 586.000000) 295th item is incorrect! (294.000000 != 588.000000) 296th item is incorrect! (295.000000 != 590.000000) 297th item is incorrect! (296.000000 != 592.000000) 298th item is incorrect! (297.000000 != 594.000000) 299th item is incorrect! (298.000000 != 596.000000) 300th item is incorrect! (299.000000 != 598.000000) 301th item is incorrect! (300.000000 != 600.000000) 302th item is incorrect! (301.000000 != 602.000000) 303th item is incorrect! (302.000000 != 604.000000) 304th item is incorrect! (303.000000 != 606.000000) 305th item is incorrect! (304.000000 != 608.000000) 306th item is incorrect! (305.000000 != 610.000000) 307th item is incorrect! (306.000000 != 612.000000) 308th item is incorrect! (307.000000 != 614.000000) 309th item is incorrect! (308.000000 != 616.000000) 310th item is incorrect! (309.000000 != 618.000000) 311th item is incorrect! (310.000000 != 620.000000) 312th item is incorrect! (311.000000 != 622.000000) 313th item is incorrect! (312.000000 != 624.000000) 314th item is incorrect! (313.000000 != 626.000000) 315th item is incorrect! (314.000000 != 628.000000) 316th item is incorrect! (315.000000 != 630.000000) 317th item is incorrect! (316.000000 != 632.000000) 318th item is incorrect! (317.000000 != 634.000000) 319th item is incorrect! (318.000000 != 636.000000) 320th item is incorrect! (319.000000 != 638.000000) 321th item is incorrect! (320.000000 != 640.000000) 322th item is incorrect! (321.000000 != 642.000000) 323th item is incorrect! (322.000000 != 644.000000) 324th item is incorrect! (323.000000 != 646.000000) 325th item is incorrect! (324.000000 != 648.000000) 326th item is incorrect! (325.000000 != 650.000000) 327th item is incorrect! (326.000000 != 652.000000) 328th item is incorrect! (327.000000 != 654.000000) 329th item is incorrect! (328.000000 != 656.000000) 330th item is incorrect! (329.000000 != 658.000000) 331th item is incorrect! (330.000000 != 660.000000) 332th item is incorrect! (331.000000 != 662.000000) 333th item is incorrect! (332.000000 != 664.000000) 334th item is incorrect! (333.000000 != 666.000000) 335th item is incorrect! (334.000000 != 668.000000) 336th item is incorrect! (335.000000 != 670.000000) 337th item is incorrect! (336.000000 != 672.000000) 338th item is incorrect! (337.000000 != 674.000000) 339th item is incorrect! (338.000000 != 676.000000) 340th item is incorrect! (339.000000 != 678.000000) 341th item is incorrect! (340.000000 != 680.000000) 342th item is incorrect! (341.000000 != 682.000000) 343th item is incorrect! (342.000000 != 684.000000) 344th item is incorrect! (343.000000 != 686.000000) 345th item is incorrect! (344.000000 != 688.000000) 346th item is incorrect! (345.000000 != 690.000000) 347th item is incorrect! (346.000000 != 692.000000) 348th item is incorrect! (347.000000 != 694.000000) 349th item is incorrect! (348.000000 != 696.000000) 350th item is incorrect! (349.000000 != 698.000000) 351th item is incorrect! (350.000000 != 700.000000) 352th item is incorrect! (351.000000 != 702.000000) 353th item is incorrect! (352.000000 != 704.000000) 354th item is incorrect! (353.000000 != 706.000000) 355th item is incorrect! (354.000000 != 708.000000) 356th item is incorrect! (355.000000 != 710.000000) 357th item is incorrect! (356.000000 != 712.000000) 358th item is incorrect! (357.000000 != 714.000000) 359th item is incorrect! (358.000000 != 716.000000) 360th item is incorrect! (359.000000 != 718.000000) 361th item is incorrect! (360.000000 != 720.000000) 362th item is incorrect! (361.000000 != 722.000000) 363th item is incorrect! (362.000000 != 724.000000) 364th item is incorrect! (363.000000 != 726.000000) 365th item is incorrect! (364.000000 != 728.000000) 366th item is incorrect! (365.000000 != 730.000000) 367th item is incorrect! (366.000000 != 732.000000) 368th item is incorrect! (367.000000 != 734.000000) 369th item is incorrect! (368.000000 != 736.000000) 370th item is incorrect! (369.000000 != 738.000000) 371th item is incorrect! (370.000000 != 740.000000) 372th item is incorrect! (371.000000 != 742.000000) 373th item is incorrect! (372.000000 != 744.000000) 374th item is incorrect! (373.000000 != 746.000000) 375th item is incorrect! (374.000000 != 748.000000) 376th item is incorrect! (375.000000 != 750.000000) 377th item is incorrect! (376.000000 != 752.000000) 378th item is incorrect! (377.000000 != 754.000000) 379th item is incorrect! (378.000000 != 756.000000) 380th item is incorrect! (379.000000 != 758.000000) 381th item is incorrect! (380.000000 != 760.000000) 382th item is incorrect! (381.000000 != 762.000000) 383th item is incorrect! (382.000000 != 764.000000) 384th item is incorrect! (383.000000 != 766.000000) 385th item is incorrect! (384.000000 != 768.000000) 386th item is incorrect! (385.000000 != 770.000000) 387th item is incorrect! (386.000000 != 772.000000) 388th item is incorrect! (387.000000 != 774.000000) 389th item is incorrect! (388.000000 != 776.000000) 390th item is incorrect! (389.000000 != 778.000000) 391th item is incorrect! (390.000000 != 780.000000) 392th item is incorrect! (391.000000 != 782.000000) 393th item is incorrect! (392.000000 != 784.000000) 394th item is incorrect! (393.000000 != 786.000000) 395th item is incorrect! (394.000000 != 788.000000) 396th item is incorrect! (395.000000 != 790.000000) 397th item is incorrect! (396.000000 != 792.000000) 398th item is incorrect! (397.000000 != 794.000000) 399th item is incorrect! (398.000000 != 796.000000) 400th item is incorrect! (399.000000 != 798.000000) 401th item is incorrect! (400.000000 != 800.000000) 402th item is incorrect! (401.000000 != 802.000000) 403th item is incorrect! (402.000000 != 804.000000) 404th item is incorrect! (403.000000 != 806.000000) 405th item is incorrect! (404.000000 != 808.000000) 406th item is incorrect! (405.000000 != 810.000000) 407th item is incorrect! (406.000000 != 812.000000) 408th item is incorrect! (407.000000 != 814.000000) 409th item is incorrect! (408.000000 != 816.000000) 410th item is incorrect! (409.000000 != 818.000000) 411th item is incorrect! (410.000000 != 820.000000) 412th item is incorrect! (411.000000 != 822.000000) 413th item is incorrect! (412.000000 != 824.000000) 414th item is incorrect! (413.000000 != 826.000000) 415th item is incorrect! (414.000000 != 828.000000) 416th item is incorrect! (415.000000 != 830.000000) 417th item is incorrect! (416.000000 != 832.000000) 418th item is incorrect! (417.000000 != 834.000000) 419th item is incorrect! (418.000000 != 836.000000) 420th item is incorrect! (419.000000 != 838.000000) 421th item is incorrect! (420.000000 != 840.000000) 422th item is incorrect! (421.000000 != 842.000000) 423th item is incorrect! (422.000000 != 844.000000) 424th item is incorrect! (423.000000 != 846.000000) 425th item is incorrect! (424.000000 != 848.000000) 426th item is incorrect! (425.000000 != 850.000000) 427th item is incorrect! (426.000000 != 852.000000) 428th item is incorrect! (427.000000 != 854.000000) 429th item is incorrect! (428.000000 != 856.000000) 430th item is incorrect! (429.000000 != 858.000000) 431th item is incorrect! (430.000000 != 860.000000) 432th item is incorrect! (431.000000 != 862.000000) 433th item is incorrect! (432.000000 != 864.000000) 434th item is incorrect! (433.000000 != 866.000000) 435th item is incorrect! (434.000000 != 868.000000) 436th item is incorrect! (435.000000 != 870.000000) 437th item is incorrect! (436.000000 != 872.000000) 438th item is incorrect! (437.000000 != 874.000000) 439th item is incorrect! (438.000000 != 876.000000) 440th item is incorrect! (439.000000 != 878.000000) 441th item is incorrect! (440.000000 != 880.000000) 442th item is incorrect! (441.000000 != 882.000000) 443th item is incorrect! (442.000000 != 884.000000) 444th item is incorrect! (443.000000 != 886.000000) 445th item is incorrect! (444.000000 != 888.000000) 446th item is incorrect! (445.000000 != 890.000000) 447th item is incorrect! (446.000000 != 892.000000) 448th item is incorrect! (447.000000 != 894.000000) 449th item is incorrect! (448.000000 != 896.000000) 450th item is incorrect! (449.000000 != 898.000000) 451th item is incorrect! (450.000000 != 900.000000) 452th item is incorrect! (451.000000 != 902.000000) 453th item is incorrect! (452.000000 != 904.000000) 454th item is incorrect! (453.000000 != 906.000000) 455th item is incorrect! (454.000000 != 908.000000) 456th item is incorrect! (455.000000 != 910.000000) 457th item is incorrect! (456.000000 != 912.000000) 458th item is incorrect! (457.000000 != 914.000000) 459th item is incorrect! (458.000000 != 916.000000) 460th item is incorrect! (459.000000 != 918.000000) 461th item is incorrect! (460.000000 != 920.000000) 462th item is incorrect! (461.000000 != 922.000000) 463th item is incorrect! (462.000000 != 924.000000) 464th item is incorrect! (463.000000 != 926.000000) 465th item is incorrect! (464.000000 != 928.000000) 466th item is incorrect! (465.000000 != 930.000000) 467th item is incorrect! (466.000000 != 932.000000) 468th item is incorrect! (467.000000 != 934.000000) 469th item is incorrect! (468.000000 != 936.000000) 470th item is incorrect! (469.000000 != 938.000000) 471th item is incorrect! (470.000000 != 940.000000) 472th item is incorrect! (471.000000 != 942.000000) 473th item is incorrect! (472.000000 != 944.000000) 474th item is incorrect! (473.000000 != 946.000000) 475th item is incorrect! (474.000000 != 948.000000) 476th item is incorrect! (475.000000 != 950.000000) 477th item is incorrect! (476.000000 != 952.000000) 478th item is incorrect! (477.000000 != 954.000000) 479th item is incorrect! (478.000000 != 956.000000) 480th item is incorrect! (479.000000 != 958.000000) 481th item is incorrect! (480.000000 != 960.000000) 482th item is incorrect! (481.000000 != 962.000000) 483th item is incorrect! (482.000000 != 964.000000) 484th item is incorrect! (483.000000 != 966.000000) 485th item is incorrect! (484.000000 != 968.000000) 486th item is incorrect! (485.000000 != 970.000000) 487th item is incorrect! (486.000000 != 972.000000) 488th item is incorrect! (487.000000 != 974.000000) 489th item is incorrect! (488.000000 != 976.000000) 490th item is incorrect! (489.000000 != 978.000000) 491th item is incorrect! (490.000000 != 980.000000) 492th item is incorrect! (491.000000 != 982.000000) 493th item is incorrect! (492.000000 != 984.000000) 494th item is incorrect! (493.000000 != 986.000000) 495th item is incorrect! (494.000000 != 988.000000) 496th item is incorrect! (495.000000 != 990.000000) 497th item is incorrect! (496.000000 != 992.000000) 498th item is incorrect! (497.000000 != 994.000000) 499th item is incorrect! (498.000000 != 996.000000) 500th item is incorrect! (499.000000 != 998.000000) 501th item is incorrect! (500.000000 != 1000.000000) 502th item is incorrect! (501.000000 != 1002.000000) 503th item is incorrect! (502.000000 != 1004.000000) 504th item is incorrect! (503.000000 != 1006.000000) 505th item is incorrect! (504.000000 != 1008.000000) 506th item is incorrect! (505.000000 != 1010.000000) 507th item is incorrect! (506.000000 != 1012.000000) 508th item is incorrect! (507.000000 != 1014.000000) 509th item is incorrect! (508.000000 != 1016.000000) 510th item is incorrect! (509.000000 != 1018.000000) 511th item is incorrect! (510.000000 != 1020.000000) 512th item is incorrect! (511.000000 != 1022.000000) 511 errors! ERROR: ACCL base functionality test failed! -- STATISTICS - ID: 0 ----------------------------------------------- Read command FIFO used: 0 Write command FIFO used: 0 Host reads sent: 98 Host writes sent: 66 Card reads sent: 65 Card writes sent: 64 Sync reads sent: 10 Sync writes sent: 0 Page faults: 0 -- [31m[1mNET STATS[0m[0m QSFP0 RX pkgs: 510 TX pkgs: 138 ARP RX pkgs: 4 ARP TX pkgs: 2 ICMP RX pkgs: 0 ICMP TX pkgs: 0 TCP RX pkgs: 0 TCP TX pkgs: 0 ROCE RX pkgs: 180 ROCE TX pkgs: 136 IBV RX pkgs: 235 IBV TX pkgs: 236 PSN drop cnt: 0 Retrans cnt: 0 TCP session cnt: 0 STRM down: 0 Finalizing MPI... Done. Terminating... ```

stderr

``` XRT build version: 2.13.466 Build hash: f5505e402c2ca1ffe45eb6d3a9399b23a0dc8776 Build date: 2022-04-14 17:43:11 Git branch: 2022.1 PID: 21422 UID: 500207 [Tue May 14 12:11:34 2024 GMT] HOST: alveo-u55c-07.inf.ethz.ch EXE: /pub/scratch/lawirz/XACCL/integrations/pytorch_ddp/accl/test/host/Coyote/accl_on_coyote [XRT] ERROR: No devices found [XRT] ERROR: No devices found [XRT] ERROR: No devices found ACLL DEBUG: aquiring cProc: targetRegion: 0, cPid: 0 ACLL DEBUG: aquiring qProc: targetRegion: 0, cPid: 2 ACLL DEBUG: aquiring qProc: targetRegion: 0, cPid: 1 CCLO HWID: 2696576574 at 0x0 CCLO source commit (first 24b): a0ba7e CCLO Capabilities: Stack type: RDMA Internal DMA:True External DMA:False Reduction:True Compression:True Kernel Streams:True Debug:False Doing a soft reset Configuring Eager RX Buffers get_device_type: coyote_device get_device_type: coyote_device CoyoteBuffer contructor called! page_size:2097152, buffer_size:64,n_pages:1 Allocation successful! Allocated buffer: 7fe2c6000000, Size: 64 calling offload: 7fe2c6000000, size: 64 get_device_type: coyote_device get_device_type: coyote_device CoyoteBuffer contructor called! page_size:2097152, buffer_size:64,n_pages:1 Allocation successful! Allocated buffer: 7fe2c5e00000, Size: 64 calling offload: 7fe2c5e00000, size: 64 Configuring Rendezvous Spare Buffers get_device_type: coyote_device get_device_type: coyote_device CoyoteBuffer contructor called! page_size:2097152, buffer_size:4194304,n_pages:2 Allocation successful! Allocated buffer: 7fe2c5a00000, Size: 4194304 calling offload: 7fe2c5a00000, size: 4194304 get_device_type: coyote_device get_device_type: coyote_device CoyoteBuffer contructor called! page_size:2097152, buffer_size:4194304,n_pages:2 Allocation successful! Allocated buffer: 7fe2c5600000, Size: 4194304 calling offload: 7fe2c5600000, size: 4194304 get_device_type: coyote_device get_device_type: coyote_device CoyoteBuffer contructor called! page_size:2097152, buffer_size:4194304,n_pages:2 Allocation successful! Allocated buffer: 7fe2c5200000, Size: 4194304 calling offload: 7fe2c5200000, size: 4194304 Configuring a communicator Configuring arithmetic Configuring collective tuning parameters CCLO configured Set timeout Set max eager size: 64 Set max rendezvous reduce size: 4194304 Accelerator ready! Communicator 0 (0x40): local rank: 0 number of ranks: 2 > rank 0 (ip 10.253.74.92:5005 ; session 0 ; max segment size 4194304) : <- inbound seq number 0, -> outbound seq number 0 > rank 1 (ip 10.253.74.96:5005 ; session 1 ; max segment size 4194304) : <- inbound seq number 0, -> outbound seq number 0 Rank 0 passed last barrier before test! CCLO address: 0 rx address: 4 Spare RX Buffer 0: address: 0x7fe2c6000000 status: ENQUEUED occupancy: 0/64 MPI tag: 0 seq: 0 src: 0 Spare RX Buffer 1: address: 0x7fe2c5e00000 status: ENQUEUED occupancy: 0/64 MPI tag: 0 seq: 0 src: 0 CoyoteBuffer contructor called! page_size:2097152, buffer_size:2048,n_pages:1 Allocation successful! Allocated buffer: 7fe2c5000000, Size: 2048 CoyoteBuffer contructor called! page_size:2097152, buffer_size:2048,n_pages:1 Allocation successful! Allocated buffer: 7fe2c4e00000, Size: 2048 Reducing data... Free user buffer from cProc cPid:0, buffer_size:2048,7fe2c5000000 Free user buffer from cProc cPid:0, buffer_size:2048,7fe2c4e00000 Communicator 0 (0x40): local rank: 0 number of ranks: 2 > rank 0 (ip 10.253.74.92:5005 ; session 0 ; max segment size 4194304) : <- inbound seq number 0, -> outbound seq number 0 > rank 1 (ip 10.253.74.96:5005 ; session 1 ; max segment size 4194304) : <- inbound seq number 0, -> outbound seq number 1 CCLO address: 0 rx address: 4 Spare RX Buffer 0: address: 0x7fe2c6000000 status: ENQUEUED occupancy: 0/64 MPI tag: 0 seq: 0 src: 0 Spare RX Buffer 1: address: 0x7fe2c5e00000 status: ENQUEUED occupancy: 0/64 MPI tag: 0 seq: 0 src: 0 Removing CCLO object at 0 Doing a soft reset Free user buffer from cProc cPid:0, buffer_size:64,7fe2c6000000 Free user buffer from cProc cPid:0, buffer_size:64,7fe2c5e00000 Free user buffer from cProc cPid:0, buffer_size:4194304,7fe2c5a00000 Free user buffer from cProc cPid:0, buffer_size:4194304,7fe2c5600000 Free user buffer from cProc cPid:0, buffer_size:4194304,7fe2c5200000 ```

Xilinx / ACCL

Reduce/Allreduce issues on cyt_rdma #196

Output of first run(allreduce):

Second run: