search

openucx / ucx

Unified Communication X (mailing list - https://elist.ornl.gov/mailman/listinfo/ucx-group)
http://www.openucx.org
Other
1.16k stars 428 forks source link

tcp/test_uct_sockaddr.err_handle/1 </lo> failure #6451

Open evgeny-leksikov opened 3 years ago

evgeny-leksikov commented 3 years ago 
[ RUN      ] tcp/test_uct_sockaddr.err_handle/1 </lo>
[     INFO ] Testing tcp on 127.0.0.1:47648 interface lo with RDMA_CM_SOURCE_ADDRESS=127.0.0.1
/scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../test/gtest/uct/ib/test_sockaddr.cc:245: Failure
wrong data length received 0
terminate called after throwing an instance of 'ucs::test_abort_exception'
  what():  std::exception
[swx-rdmz-ucx-new-02:29713:async:12892] Caught signal 6 (Aborted: tkill(2) or tgkill(2))
==== backtrace (tid:  12892) ====
 0 0x0000000000036207 __GI_raise()  :0
 1 0x00000000000378f8 __GI_abort()  :0
 2 0x00000000000607d5 __gnu_cxx::__verbose_terminate_handler()  ???:0
 3 0x000000000005e746 std::rethrow_exception()  ???:0
 4 0x000000000005e773 std::terminate()  ???:0
 5 0x000000000005e993 __cxa_throw()  ???:0
 6 0x0000000000a96bd1 test_uct_sockaddr::verify_remote_data()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../test/gtest/uct/ib/test_sockaddr.cc:245
 7 0x0000000000a9773c test_uct_sockaddr::client_connect_cb()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../test/gtest/uct/ib/test_sockaddr.cc:358
 8 0x0000000000015445 uct_cm_ep_client_connect_cb()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/uct/base/uct_cm.c:109
 9 0x000000000002db06 uct_tcp_sockcm_ep_client_connect_cb()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/uct/tcp/tcp_sockcm_ep.c:52
10 0x000000000002db06 uct_tcp_sockcm_ep_client_invoke_connect_cb()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/uct/tcp/tcp_sockcm_ep.c:555
11 0x000000000002db06 uct_tcp_sockcm_ep_handle_data_received()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/uct/tcp/tcp_sockcm_ep.c:612
12 0x000000000002de50 uct_tcp_sockcm_ep_recv()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/uct/tcp/tcp_sockcm_ep.c:693
13 0x000000000002b768 uct_tcp_sa_data_handler()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/uct/tcp/tcp_sockcm.c:112
14 0x000000000004753a ucs_async_handler_invoke()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucs/async/async.c:249
15 0x000000000004753a ucs_async_dispatch_handlers()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucs/async/async.c:303
16 0x000000000004ae6c ucs_async_thread_ev_handler()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucs/async/thread.c:87
17 0x000000000006b221 ucs_event_set_wait()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucs/sys/event_set.c:215
18 0x000000000004b6f3 ucs_async_thread_func()  /scrap/azure/agent-04/AZP_WORKSPACE/2/s/contrib/../src/ucs/async/thread.c:130
19 0x0000000000007dd5 start_thread()  pthread_create.c:0
20 0x00000000000fdead __clone()  ???:0
=================================
[swx-rdmz-ucx-new-02:29713:async:12892] Process frozen...
make: *** [test] Terminated
##[error]Bash exited with code '124'.

full log https://dev.azure.com/ucfconsort/0b36e3f0-8ab9-4a48-b68b-4b2350e02c88/_apis/build/builds/14198/logs/298 occurred here https://github.com/openucx/ucx/pull/6427

alinask commented 3 years ago

Ran on the same host 8.5 hours in a row without reproduction.