search

IIC-SIG-MLsys / HDDT

Distrubuted DNN Training on Heterogeneous GPUs
0 stars 6 forks source link

Add support to MLU #27

Closed derekwin closed 1 week ago

derekwin commented 1 week ago

finished by https://github.com/IIC-SIG-MLsys/HDDT/pull/29

derekwin commented 1 week ago

using hygon(amd) gpu as client

./rdma_client 
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1113 21:13:21.538247 3625843 driver.cpp:92] Using ROCm Device with ID: 1, Name: Z100L, PCI Bus ID: 0xa3, GCN Arch: 21943.
W1113 21:13:21.538390 3625843 net.h:185] init sockaddr.
W1113 21:13:21.538440 3625843 net.h:198] start alloc buffer.
I1113 21:13:21.538455 3625843 rocm_memory.cpp:23] Allocate memory using hipMalloc.
W1113 21:13:21.560113 3625843 net.h:209] setup client.
W1113 21:13:21.560731 3625843 rdma_communictor.cpp:352] Client: RDMA CM event channel is created at : 0x5588b3b469d0.
W1113 21:13:21.560786 3625843 rdma_communictor.cpp:371] waiting for cm event: RDMA_CM_EVENT_ADDR_RESOLVED.
W1113 21:13:21.584888 3625843 rdma_communictor.cpp:680] A new RDMA_CM_EVENT_ADDR_RESOLVED type event is received 
W1113 21:13:21.584903 3625843 rdma_communictor.cpp:385] RDMA address is resolved.
W1113 21:13:21.584973 3625843 rdma_communictor.cpp:393] waiting for cm event: RDMA_CM_EVENT_ROUTE_RESOLVED.
W1113 21:13:21.584990 3625843 rdma_communictor.cpp:680] A new RDMA_CM_EVENT_ROUTE_RESOLVED type event is received 
I1113 21:13:21.585002 3625843 rdma_communictor.cpp:408] Trying to connect to server at : 192.168.2.252 port: 2024 
W1113 21:13:21.585115 3625843 rdma_communictor.cpp:751] rdma_buffer_register: pd is 0x5588b3b6a2f0, addr is 0x7f53b9400000, len is 1024, permission is 7.
W1113 21:13:21.585469 3625843 rdma_communictor.cpp:758] Registered: 0x7f53b9400000 , len: 1024 , stag: 0x204dd1 
I1113 21:13:21.585479 3625843 rdma_communictor.cpp:424] rdma_buffer_register client_send_buffer_mr to pd
W1113 21:13:21.585490 3625843 rdma_communictor.cpp:751] rdma_buffer_register: pd is 0x5588b3b6a2f0, addr is 0x7f53b9400000, len is 1024, permission is 7.
W1113 21:13:21.585810 3625843 rdma_communictor.cpp:758] Registered: 0x7f53b9400000 , len: 1024 , stag: 0x204ed2 
I1113 21:13:21.585825 3625843 rdma_communictor.cpp:433] rdma_buffer_register client_recv_buffer_mr to pd
I1113 21:13:21.585832 3625843 rdma_communictor.cpp:438] Client prepare memory region success.
W1113 21:13:21.585851 3625843 rdma_communictor.cpp:448] completion event channel created at : 0x5588b3b48490 
W1113 21:13:21.586210 3625843 rdma_communictor.cpp:457] CQ created at 0x5588b3b6a3a0 with 31 elements.
W1113 21:13:21.586222 3625843 rdma_communictor.cpp:465] Create qp 0x5588b3b15220.
W1113 21:13:21.586238 3625843 rdma_communictor.cpp:482] cap.max_recv_sge is 2.
W1113 21:13:21.586807 3625843 rdma_communictor.cpp:490] QP created at 0x5588b3b6d458 
W1113 21:13:21.586820 3625843 rdma_communictor.cpp:751] rdma_buffer_register: pd is 0x5588b3b6a2f0, addr is 0x5588b3b15190, len is 16, permission is 1.
W1113 21:13:21.586853 3625843 rdma_communictor.cpp:758] Registered: 0x5588b3b15190 , len: 16 , stag: 0x1ffbba 
W1113 21:13:21.586862 3625843 rdma_communictor.cpp:500] Setup the newserver metadata mr is successful
W1113 21:13:21.586877 3625843 rdma_communictor.cpp:648] Post request work successful.
W1113 21:13:21.586891 3625843 rdma_communictor.cpp:512] Pre-post receive newserver metadata is successful
W1113 21:13:21.586930 3625843 rdma_communictor.cpp:532] Waiting for cm event: RDMA_CM_EVENT_ESTABLISHED.
W1113 21:13:21.622901 3625843 rdma_communictor.cpp:680] A new RDMA_CM_EVENT_ESTABLISHED type event is received 
I1113 21:13:21.622915 3625843 rdma_communictor.cpp:544] The client is connected successfully.
I1113 21:13:21.622925 3625843 rdma_communictor.cpp:548] Start xchange
W1113 21:13:21.622934 3625843 rdma_communictor.cpp:751] rdma_buffer_register: pd is 0x5588b3b6a2f0, addr is 0x5588b3b151a0, len is 16, permission is 1.
W1113 21:13:21.622985 3625843 rdma_communictor.cpp:758] Registered: 0x5588b3b151a0 , len: 16 , stag: 0x1ffdbc 
W1113 21:13:21.623005 3625843 rdma_communictor.cpp:648] Post request work successful.
W1113 21:13:21.623248 3625843 rdma_communictor.cpp:721] 2 WC are completed.
W1113 21:13:21.623262 3625843 rdma_communictor.cpp:585] Server sent us its buffer location and credentials, showing 
I1113 21:13:21.623277 3625843 rdma_communictor.cpp:777] ---------------------------------------------------------
I1113 21:13:21.623286 3625843 rdma_communictor.cpp:780] buffer attr, addr: 0x30080fff8000000 , len: 1024 , stag : 0x23fe30
I1113 21:13:21.623296 3625843 rdma_communictor.cpp:781] ---------------------------------------------------------
client Write Data: Hello World!
W1113 21:13:21.623418 3625843 rdma_communictor.cpp:648] Post request work successful.
W1113 21:13:21.623433 3625843 rdma_communictor.cpp:721] 1 WC are completed.
W1113 21:13:21.623440 3625843 rdma_communictor.cpp:65] Client side WRITE is complete.

using mlu as server

./rdma_server 
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1113 21:12:35.529850 2601258 server.cpp:21] 0x562903e3c6a0
W1113 21:12:35.529973 2601258 net.h:185] init sockaddr.
W1113 21:12:35.529994 2601258 net.h:198] start alloc buffer.
I1113 21:12:35.530006 2601258 neuware_memory.cpp:20] Allocate memory using cnMalloc.
W1113 21:12:35.530154 2601258 net.h:205] setup server.
W1113 21:12:35.530675 2601258 rdma_communictor.cpp:108] Server: RDMA CM event channel is created successfully at 0x562903e9b9c0.
W1113 21:12:35.530694 2601258 rdma_communictor.cpp:118] RDMA connection id for the server is created.
W1113 21:12:35.530706 2601258 rdma_communictor.cpp:127] Server RDMA CM id is successfully binded.
I1113 21:12:35.530727 2601258 rdma_communictor.cpp:145] Server is listening successfully at: 0.0.0.0 , port: 2024.
W1113 21:13:21.628855 2601258 rdma_communictor.cpp:679] A new RDMA_CM_EVENT_CONNECT_REQUEST type event is received 
W1113 21:13:21.628911 2601258 rdma_communictor.cpp:162] A new RDMA client(newconnection) connection id is stored at 0x562903ea0090.
W1113 21:13:21.629040 2601258 rdma_communictor.cpp:173] A new protection domain is allocated at 0x562903ea0580.
W1113 21:13:21.629058 2601258 rdma_communictor.cpp:749] rdma_buffer_register: pd is 0x562903ea0580, addr is 0x30080fff8000000, len is 1024, permission is 7.
W1113 21:13:21.629513 2601258 rdma_communictor.cpp:757] Registered: 0x30080fff8000000 , len: 1024 , stag: 0x23fd2f 
W1113 21:13:21.629532 2601258 rdma_communictor.cpp:749] rdma_buffer_register: pd is 0x562903ea0580, addr is 0x30080fff8000000, len is 1024, permission is 7.
W1113 21:13:21.629905 2601258 rdma_communictor.cpp:757] Registered: 0x30080fff8000000 , len: 1024 , stag: 0x23fe30 
I1113 21:13:21.629921 2601258 rdma_communictor.cpp:196] Server prepare memory region success.
W1113 21:13:21.629942 2601258 rdma_communictor.cpp:205] An I/O completion event channel is created at 0x562903e3c720.
W1113 21:13:21.630400 2601258 rdma_communictor.cpp:218] Completion queue (CQ) is created at 0x562903ea0680 with 31 elements.
W1113 21:13:21.631170 2601258 rdma_communictor.cpp:248] Newconnection QP created at 0x562903ea0988.
W1113 21:13:21.631188 2601258 rdma_communictor.cpp:251] Waiting to accept a new connection.
W1113 21:13:21.631203 2601258 rdma_communictor.cpp:749] rdma_buffer_register: pd is 0x562903ea0580, addr is 0x562903e9ac60, len is 16, permission is 1.
W1113 21:13:21.631254 2601258 rdma_communictor.cpp:757] Registered: 0x562903e9ac60 , len: 16 , stag: 0x23b2b1 
W1113 21:13:21.631273 2601258 rdma_communictor.cpp:648] Post request work successful.
W1113 21:13:21.631287 2601258 rdma_communictor.cpp:815] Receive buffer pre-posting is successful.
W1113 21:13:21.631865 2601258 rdma_communictor.cpp:828] Going to wait for : RDMA_CM_EVENT_ESTABLISHED event.
W1113 21:13:21.632506 2601258 rdma_communictor.cpp:679] A new RDMA_CM_EVENT_ESTABLISHED type event is received 
A new connection is accepted from 192.168.2.241 
W1113 21:13:21.632558 2601258 rdma_communictor.cpp:259] Start to send metadata to the new connection.
W1113 21:13:21.632607 2601258 rdma_communictor.cpp:721] 1 WC are completed.
I1113 21:13:21.632632 2601258 rdma_communictor.cpp:865] Client side buffer information is received...
I1113 21:13:21.632647 2601258 rdma_communictor.cpp:777] ---------------------------------------------------------
I1113 21:13:21.632661 2601258 rdma_communictor.cpp:778] buffer attr, addr: 0x7f53b9400000 , len: 1024 , stag : 0x204dd1
I1113 21:13:21.632674 2601258 rdma_communictor.cpp:781] ---------------------------------------------------------
I1113 21:13:21.632687 2601258 rdma_communictor.cpp:867] The client has requested buffer length of : 1024 bytes 
W1113 21:13:21.632701 2601258 rdma_communictor.cpp:749] rdma_buffer_register: pd is 0x562903ea0580, addr is 0x562903e9ac70, len is 16, permission is 1.
W1113 21:13:21.632766 2601258 rdma_communictor.cpp:757] Registered: 0x562903e9ac70 , len: 16 , stag: 0x239e9d 
W1113 21:13:21.632797 2601258 rdma_communictor.cpp:648] Post request work successful.
W1113 21:13:21.632824 2601258 rdma_communictor.cpp:721] 1 WC are completed.
W1113 21:13:21.632849 2601258 rdma_communictor.cpp:902] Local buffer metadata has been sent to the client.
Server get Data: Hello World!