Closed mikehb closed 10 months ago
HomaModule's homa_test couldn't connect the remote server too
root@hb1:~/HomaModule/util# ./homa_test 10.212.155.216:4000 stream
Count too large; reducing from 1000 to 100
Error in recvmsg: Connection timed out
root@hb1:~/HomaModule/util#
root@hb:~/HomaModule/util# ./server
root@hb:~/HomaModule/util# netstat -nap | grep server
Active Internet connections (servers and established)
tcp 0 0 0.0.0.0:4000 0.0.0.0:* LISTEN 110254/./server
Active UNIX domain sockets (servers and established)
root@hb:~/HomaModule/util#
Strange: all of these tests work fine for me, both locally and remotely. I wonder if there might be a problem in delivering Homa packets over your network (e.g., perhaps they are getting dropped by the switch, or perhaps Homa is trying to use TSO but your NICs refuse to do TSO for Homa packets and just drop them)? How about trying a simple Homa test:
-John-
On Mon, Oct 16, 2023 at 4:02 AM mikehb @.***> wrote:
HomaModule's homa_test couldn't connect the remote server too
@.:~/HomaModule/util# ./homa_test 10.212.155.216:4000 stream Count too large; reducing from 1000 to 100 Error in recvmsg: Connection timed out @.:~/HomaModule/util#
@.***:~/HomaModule/util# ./server
@.:~/HomaModule/util# netstat -nap | grep server Active Internet connections (servers and established) tcp 0 0 0.0.0.0:4000 0.0.0.0: LISTEN 110254/./server Active UNIX domain sockets (servers and established) **@.***:~/HomaModule/util#
— Reply to this email directly, view it on GitHub https://github.com/PlatformLab/grpc_homa/issues/13#issuecomment-1764228717, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOOUCUD7VSLQL2LIGB4OTLX7UH25AVCNFSM6AAAAAA6BQ5H2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRUGIZDQNZRG4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>
I execute cp_node server/client on the same machine, but it prints following error. Is the cp_node using fixed node name? Could cp_node add an option to take a server ip address?
root@hb:~/HomaModule/util# hostname
hb
root@hb:~/HomaModule/util# ./cp_node client
1697682117.885167844 FATAL: couldn't look up address for node1: Temporary failure in name resolution
root@hb:~/HomaModule/util#
I modified the code to use server's IP, and cp_node still print "Connection timed out"
/**
* init_server_addrs() - Set up the server_addrs table (addresses of the
* server/port combinations that clients will communicate with), based on
* current configuration parameters. Any previous contents of the table
* are discarded. This also initializes related arrays @server_ids and
* @freeze.
*/
void init_server_addrs(void)
{
server_addrs.clear();
server_conns.clear();
freeze.clear();
first_id.clear();
for (int node: server_ids) {
char host[100];
struct addrinfo hints;
struct addrinfo *matching_addresses;
sockaddr_in_union *dest;
if (node == id)
continue;
//snprintf(host, sizeof(host), "node%d", node);
snprintf(host, sizeof(host), "%s", "10.212.155.216");
memset(&hints, 0, sizeof(struct addrinfo));
hints.ai_family = inet_family;
hints.ai_socktype = SOCK_DGRAM;
int status = getaddrinfo(host, NULL, &hints,
&matching_addresses);
if (status != 0) {
log(NORMAL, "FATAL: couldn't look up address "
"for %s: %s\n",
host, gai_strerror(status));
exit(1);
}
dest = reinterpret_cast<sockaddr_in_union *>
(matching_addresses->ai_addr);
while (((int) first_id.size()) < node)
first_id.push_back(-1);
first_id.push_back((int) server_addrs.size());
for (int thread = 0; thread < server_ports; thread++) {
dest->in4.sin_port = htons(first_port + thread);
server_addrs.push_back(*dest);
server_conns.emplace_back(node, thread, id, 0);
}
while (((int) freeze.size()) <= node)
freeze.push_back(0);
freeaddrinfo(matching_addresses);
}
}
root@hb1:~/HomaModule/util# ./cp_node client
1697683451.762402699 Average message length 0.1 KB, rate 0.00 K/sec, expected BW 0.0 Gbps
1697683451.825312410 FATAL: error in recvmsg: Connection timed out (id 0, server Unknown family 0)
root@hb1:~/HomaModule/util#
"ideally while the test is still running and failing", the cp_node client exist when connection failed node_client.txt node_server.txt
I took a look at the traces but unfortunately the client trace is completely empty. That doesn't make sense to me: at the least there should be invocations of Homa to send a message, timer firings, and so on. Can you try to collect it again? The server trace is nearly empty, but plausible: the application invokes recvmsg but no packets seem to arrive.
-John-
On Wed, Oct 18, 2023 at 7:58 PM mikehb @.***> wrote:
"ideally while the test is still running and failing", the cp_node client exist when connection failed node_client.txt https://github.com/PlatformLab/grpc_homa/files/13037142/node_client.txt node_server.txt https://github.com/PlatformLab/grpc_homa/files/13037144/node_server.txt
— Reply to this email directly, view it on GitHub https://github.com/PlatformLab/grpc_homa/issues/13#issuecomment-1769813411, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOOUCX6QZUPRO4OA4KSAITYACJL5AVCNFSM6AAAAAA6BQ5H2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRZHAYTGNBRGE . You are receiving this because you commented.Message ID: @.***>
root@hb1:~/HomaModule# lsmod | grep homa
homa 294912 0
root@hb1:~/HomaModule# cd util/
root@hb1:~/HomaModule/util# ./cp_node client
1697697036.039309767 Average message length 0.1 KB, rate 0.00 K/sec, expected BW 0.0 Gbps
1697697036.105974125 FATAL: error in recvmsg: Connection timed out (id 0, server Unknown family 0)
root@hb1:~/HomaModule/util# ll /proc/time
timer_list timetrace
root@hb1:~/HomaModule/util# cat /proc/timetrace
cat: /proc/timetrace: Bad address
root@hb1:~/HomaModule/util# ./ttprint.py
root@hb1:~/HomaModule/util#
node_client.txt node_server.txt
execute following script in on terminal
for (( ;; )) do ./cp_node client; sleep 1; done
and catch the trace on another terminal
./ttprint.py > node_client.txt
./ttprint.py is empty since there is not client connection. I got the above node_server.txt by executing './cp_node client' in the same host
root@hb:~# ethtool -k ens3 | grep -i tcp-segmentation-offload
tcp-segmentation-offload: on
root@hb:~#
I'm not sure I understand what's happening with your attempts to get client-side timetraces.
First, reading the timetrace will clear the buffer, so if you invoke "cat /proc/timetrace" and then invoke ttprint.py, ttprint.py will return nothing unless there has been additional Homa activity since the "cat /proc/timetrace".
I don't understand your comment "./ttprint.py is empty since there is not client connection": if you ran "cp_node client" a few times, which appears to be the case, there should be buffered timetrace records (the buffer is only cleared when the timetrace is read).
From the latest node_client.txt file and the node_server.txt file you sent yesterday, it seems that Homa packets are being transmitted successfully but they are not being received. This suggests that they are getting dropped somewhere in the network. What is the environment in which you are running your experiments? Do you control the switch? I haven't tried running Homa in cloud providers like AWS; if you are running there, perhaps the cloud provider refuses to transmit packets that use unknown protocols?
Since this test only sends small packets, there shouldn't be any issues with TCP segmentation offload.
-John-
On Thu, Oct 19, 2023 at 3:38 AM mikehb @.***> wrote:
@.:~# ethtool -k ens3 | grep -i tcp-segmentation-offload tcp-segmentation-offload: on @.:~#
— Reply to this email directly, view it on GitHub https://github.com/PlatformLab/grpc_homa/issues/13#issuecomment-1770541373, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOOUCRTFVDREXB2YG36JN3YAD7LFAVCNFSM6AAAAAA6BQ5H2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZQGU2DCMZXGM . You are receiving this because you commented.Message ID: @.***>
I am using Ubuntu running on openstack
I use Ubuntu, so I'm pretty sure that's not the problem. I've never used openstack; is there some way you can find out how it deals with exotic network protocols? (Homa uses IP protocol 0xFD, which is defined as "Use for experimentation and testing"). You could also try changing the definition of IPPROTO_HOMA in homa.h to see if different values work (I don't have any suggestions for other values to try, though).
-John-
On Thu, Oct 19, 2023 at 7:48 PM mikehb @.***> wrote:
I am using Ubuntu running on openstack
— Reply to this email directly, view it on GitHub https://github.com/PlatformLab/grpc_homa/issues/13#issuecomment-1771992938, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOOUCXNUSMA2OEPZIPBIFLYAHQ7LAVCNFSM6AAAAAA6BQ5H2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZRHE4TEOJTHA . You are receiving this because you commented.Message ID: @.***>
I switched to VMware Workstation 17, it works for me, cp_node client could connect remote cp_node server. Maybe something wrong with Openstack, let me take a further investigation.
Network administrator disabled the network filter, and I could connect remote grpc_homa server now
Excellent; glad to hear it!
homa.tar.gz This client could connect local homa server, but couldn't connect remote homa server. Meanwhile, it could connect remote tcp server
Ubuntu 22.04.2 LTS with linux kernel 6.1.40-060140
run grpc_homa server on 10.212.155.216
run grpc_homa client on 10.212.155.195