Getting started with LKL

[speedingdaemon]

I am trying to write a hello-world client server application using LKL. Is there any documentation that I can refer to to get going? I am a bit confused on how can I use LKL to write a client/server async application. Even a sample program will help. Also, I understand that instead of my application making direct syscalls, I need to use lklsys variants of those. Does anyone know what .h file do I need to include and what .so file I need to link my app to?

I added these 2 lines to my makefile: LKL_LIB = $(LKL_DIR)/tools/lkl/ LIBS_LKL = -llkl -L$(LKL_LIB)

But I am getting a whole bunch of these undefined symbol errors: ...: undefined reference to lkl_sys_socket' ...: undefined reference tolkl_sys_setsockopt' ...:

Any help is appreciated.

thanks

[tavip]

Hi,

The easiest way to build you lkl app is to add it in tools/lkl and modify the Build file to include you app.

If you want to build you app outside tools/lkl you can use the following Makefile:

LKL_DIR = /home/tavi/src/linux/tools/lkl
CFLAGS := -I$(LKL_DIR)/include -DCONFIG_AUTO_LKL_POSIX_HOST
LDFLAGS := -L$(LKL_DIR)
LDLIBS := -llkl

net-test: net-test.o

Note: -llkl needs to be assigned to LDLIBS so that it is added to the end of the linker command, otherwise you get the undefined references issues you run into.

[speedingdaemon]

Thanks. Will this work even though I don't see the symbols such as lkl_sys_socket in liblkl.so? Sorry, asking too basic questions.

Also where can I get info on how to setup the client server app code? For example, do I need to setup some virtio device for communication to take place? How do I setup the lkl_start_process/etc? I want to run my server and client on 2 different machines.

[tavip]

lklsys* calls are inline wrappers defined in headers that call lkl_syscall.

Please take a look at tools/lkl/net-test.c for an example of using the networking part of LKL.

[speedingdaemon]

Sounds like I am still doing something wrong (still getting undefined reference error): gcc -o build/x86/linux/test_lkl build/x86/linux/test_lkl.o -g -O3 -I./../../../../../../lkl/linux//tools/lkl//include -DCONFIG_AUTO_LKL_POSIX_HOST-llkl -L./../../../../../../lkl/linux//tools/lkl/

[tavip]

-llkl seems glued to -DCONFIG_ but I am not sure if that is not a paste error. Could you please share the Makefile you are using?

[speedingdaemon]

CC=gcc CFLAGS=-g -O3 CFLAGS_SSL=$(CFLAGS) -DTLSTEST_SSL_ENABLE

SRC = src CMN_SRC = src/common LNX_SRC = src/linux

BLD = build

OPENSSL_DIR = ./third-party/openssl/ OPENSSL_INC = $(OPENSSL_DIR)/include/ OPENSSL_LIB = $(OPENSSL_DIR)/build/${ARCH}/linux/latest/ LKL_DIR = ./../../../../../../lkl/linux/ LKL_LIB = $(LKL_DIR)/tools/lkl/ INC = -I$(OPENSSL_INC) -I$(CMN_SRC)/ LIBS = -lpthread LIBS_SSL = -lssl -lcrypto -lpthread -L$(OPENSSL_LIB)

CFLAGS_LKL := $(CFLAGS) -I$(LKL_LIB)/include -DCONFIG_AUTO_LKL_POSIX_HOST LDFLAGS := -L$(LKL_LIB) LDLIBS := -llkl

UNAME_P := $(shell uname -p) ifeq ($(UNAME_P), x86_64) ARCH=x86 else ARCH=arm endif

_DEPS=tlstest_common.h DEPS = $(patsubst %,$(CMN_SRC)/%,$(_DEPS))

OBJDIR=$(BLD)/$(ARCH)/linux

SSL_SERVER_LKL=$(OBJDIR)/tlstest_server_ssl_lkl SSL_CLIENT_LKL=$(OBJDIR)/tlstest_client_ssl_lkl

$(SSL_SERVER_LKL) $(SSL_CLIENT_LKL) all: $(SSL_SERVER_LKL) $(SSL_CLIENT_LKL)

$(OBJDIR)/%_ssl_lkl.o: $(LNX_SRC)/%.c $(DEPS) $(CC) -c -o $@ $< $(CFLAGS_SSL) $(INC)

$(SSL_SERVER_LKL): $(OBJDIR)/tlstest_server_ssl_lkl.o $(CC) -o $@ $^ $(CFLAGS_LKL) $(LIBS_SSL) $(LDLIBS) $(LDFLAGS)

$(SSL_CLIENT_LKL): $(OBJDIR)/tlstest_client_ssl_lkl.o $(CC) -o $@ $^ $(CFLAGS_SSL) $(LIBS_SSL) clean: @rm -rfv $(OBJDIR)

[tavip]

It looks ok. Can you give a try with the Makefile I've mentioned above and with net-test in some out of tree directory? To rule out any setup issues.

[speedingdaemon]

This worked. Wonder what mine is doing wrong.

root@srv12:~/net-test# cat Makefile LKL_DIR = ~/lkl/linux/tools/lkl CFLAGS := -I$(LKL_DIR)/include -DCONFIG_AUTO_LKL_POSIX_HOST LDFLAGS := -L$(LKL_DIR) LDLIBS := -llkl -lpthread -lrt

net-test: net-test.o server2:~/net-test# ls Makefile net-test net-test.c net-test.o test.h server2:~/net-test#

[speedingdaemon]

So now I have taken code from this (https://github.com/lkl/linux/issues/133) thread and modified it to use lklsys APIs directly on the server side: root@myserver:/lkl-test-client-server# more server.c

include <sys/socket.h>

include <netinet/in.h>

include <arpa/inet.h>

include

include

include

include

include

include <sys/types.h>

include

include

int main(int argc, char* argv) { int listenfd = 0; int connfd = 0; struct sockaddr_in serv_addr; char sendBuff[1025]; int numrv; int totalbyteswritten = 0; int byteswritten = 0; struct lkl_netdev nd = NULL; int ret, nd_id = -1, nd_ifindex = -1;

nd = lkl_netdev_tap_create("lkl_ptt1", 0);
if (!nd) {
    fprintf(stderr, "init netdev failed\n");
    return -1;
}
ret = lkl_netdev_add(nd, NULL);
if (ret < 0) {
    fprintf(stderr, "failed to add netdev: %s\n",
            lkl_strerror(ret));
}
nd_id = ret;
ret = lkl_start_kernel(&lkl_host_ops, "mem=16M loglevel=8");
if (ret) {
    fprintf(stderr, "can't start kernel: %s\n", lkl_strerror(ret));
    return -1;
}

lkl_if_up(1);

if (nd_id >= 0) {
    nd_ifindex = lkl_netdev_get_ifindex(nd_id);
    if (nd_ifindex > 0)
        lkl_if_up(nd_ifindex);
    else
        fprintf(stderr, "failed to get ifindex for netdev id %d: %s\n",
                nd_id, lkl_strerror(nd_ifindex));
}

if (nd_ifindex >= 0) {
    unsigned int addr = inet_addr("192.168.70.99");
    int nmlen = atoi("22");

    if (addr != INADDR_NONE && nmlen > 0 && nmlen < 32) {
        ret = lkl_if_set_ipv4(nd_ifindex, addr, nmlen);
        if (ret < 0)
            fprintf(stderr, "failed to set IPv4 address: %s\n",
                    lkl_strerror(ret));
    }
}

if (nd_ifindex >= 0) {
    unsigned int addr = inet_addr("192.168.68.1");

    if (addr != INADDR_NONE) {
        ret = lkl_set_ipv4_gateway(addr);
        if (ret < 0)
            fprintf(stderr, "failed to set IPv4 gateway: %s\n",
                    lkl_strerror(ret));
    }
}

listenfd = lkl_sys_socket(AF_INET, SOCK_STREAM, 0);

memset(&serv_addr, '0', sizeof(serv_addr));
memset(sendBuff, '0', sizeof(sendBuff));

serv_addr.sin_family = AF_INET;
serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
serv_addr.sin_port = htons(9600);

while (1) {
    if (lkl_sys_bind(listenfd, (struct lkl_sockaddr*)&serv_addr, sizeof(serv_addr)) != 0 ) {
        printf("BIND: Failed to bind to port. Waiting....\n");
        sleep(10);
    } else {
        printf("BIND: Successful --- Serving Port:%d.\n", serv_addr.sin_port);
        break;
    }
}

if(lkl_sys_listen(listenfd, 10) == -1)
{
    printf("Failed to listen\n");
    return -1;
}
while(1)
{
    connfd = lkl_sys_accept(listenfd, (struct lkl_sockaddr*)NULL ,NULL);

    FILE *fp = fopen(argv[1],"rb");
    if(fp==NULL)
    {
        printf("File opern error");
        return 1;
    }

    while(1)
    {
        unsigned char buff[256]={0};
        int nread = fread(buff,1,256,fp);

        if(nread > 0)
        {
            byteswritten = lkl_sys_write(connfd, buff, nread);
            totalbyteswritten += byteswritten;
            printf("totalbyteswritten so far = {%d}\n", totalbyteswritten);
        }

        if (nread < 256)
        {
            if (feof(fp))
                printf("End of file\n");
            if (ferror(fp))
                printf("Error reading\n");
            break;
        }
    }
    printf("Total bytes sent = {%d}\n", totalbyteswritten);
    totalbyteswritten = 0;
    lkl_sys_close(connfd);
    sleep(1);
}
return 0;

}

I also created the lkl_ptt1 tap interface as follows: sudo ip tuntap del dev lkl_ptt1 mode tap sudo ip tuntap add dev lkl_ptt1 mode tap user root sudo ip link set dev lkl_ptt1 up sudo ip addr add dev lkl_ptt1 sudo ip addr add dev lkl_ptt1 192.168.70.99/22

All the above, I did one my server blade.

Here are the logs when I bring up the server: ./server out [ 0.000000] Linux version 4.11.0+ (root@srv12) (gcc version 6.3.0 20170406 (Ubuntu 6.3.0-12ubuntu2) ) #3 Mon Jul 10 15:47:46 PDT 2017 [ 0.000000] bootmem address range: 0x7fe2c0000000 - 0x7fe2c3fff000 [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 16159 [ 0.000000] Kernel command line: ip=tcp virtio_mmio.device=268@0x1000000:1 [ 0.000000] PID hash table entries: 256 (order: -1, 2048 bytes) [ 0.000000] Dentry cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.000000] Inode-cache hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] Memory available: 64492k/0k RAM [ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] NR_IRQS:4096 [ 0.000000] lkl: irqs initialized [ 0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000001] lkl: time and timers initialized (irq2) [ 0.000020] pid_max: default: 4096 minimum: 301 [ 0.000053] Mount-cache hash table entries: 512 (order: 0, 4096 bytes) [ 0.000056] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes) [ 0.008266] console [lkl_console0] enabled [ 0.008298] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns [ 0.008312] xor: automatically using best checksumming function 8regs
[ 0.008454] NET: Registered protocol family 16 [ 0.172469] raid6: int64x1 gen() 2383 MB/s [ 0.344703] raid6: int64x1 xor() 1485 MB/s [ 0.516859] raid6: int64x2 gen() 3007 MB/s [ 0.689008] raid6: int64x2 xor() 2626 MB/s [ 0.861217] raid6: int64x4 gen() 3260 MB/s [ 1.034395] raid6: int64x4 xor() 2525 MB/s [ 1.206356] raid6: int64x8 gen() 3484 MB/s [ 1.378440] raid6: int64x8 xor() 2353 MB/s [ 1.378443] raid6: using algorithm int64x8 gen() 3484 MB/s [ 1.378445] raid6: .... xor() 2353 MB/s, rmw enabled [ 1.378493] raid6: using intx1 recovery algorithm [ 1.379917] clocksource: Switched to clocksource lkl [ 1.380331] NET: Registered protocol family 2 [ 1.380643] TCP established hash table entries: 512 (order: 0, 4096 bytes) [ 1.380648] TCP bind hash table entries: 512 (order: 0, 4096 bytes) [ 1.380658] TCP: Hash tables configured (established 512 bind 512) [ 1.381195] UDP hash table entries: 128 (order: 0, 4096 bytes) [ 1.381198] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes) [ 1.381431] virtio-mmio: Registering device virtio-mmio.0 at 0x1000000-0x100010b, IRQ 1. [ 1.382726] workingset: timestamp_bits=62 max_order=14 bucket_order=0 [ 1.383970] SGI XFS with ACLs, security attributes, no debug enabled [ 1.388767] io scheduler noop registered [ 1.388774] io scheduler deadline registered [ 1.388802] io scheduler cfq registered (default) [ 1.388804] io scheduler mq-deadline registered [ 1.388877] virtio-mmio virtio-mmio.0: Failed to enable 64-bit or 32-bit DMA. Trying to continue, but this might not work. [ 1.392174] NET: Registered protocol family 10 [ 1.392617] Segment Routing with IPv6 [ 1.392632] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver [ 1.393319] Btrfs loaded, crc32c=crc32c-generic [ 1.420110] IP-Config: Guessing netmask 255.255.255.0 [ 1.420118] IP-Config: Complete: [ 1.420121] device=eth0, hwaddr=ea:d2:db:db:5e:a3, ipaddr=206.0.0.0, mask=255.255.255.0, gw=255.255.255.255 [ 1.420139] host=206.0.0.0, domain=, nis-domain=(none) [ 1.420141] bootserver=255.255.255.255, rootserver=255.255.255.255, rootpath= [ 1.420214] Warning: unable to open an initial console. [ 1.420239] This architecture does not have kernel memory protection. BIND: Successful --- Serving Port:32805. [ 2.540044] random: fast init done [ 133.669991] random: crng init done

On another machine where I am running my client, I am able to ping the above IP: ping 192.168.70.99 PING 192.168.70.99 (192.168.70.99) 56(84) bytes of data. 64 bytes from 192.168.70.99: icmp_seq=1 ttl=64 time=0.203 ms 64 bytes from 192.168.70.99: icmp_seq=2 ttl=64 time=0.218 ms 64 bytes from 192.168.70.99: icmp_seq=3 ttl=64 time=0.241 ms 64 bytes from 192.168.70.99: icmp_seq=4 ttl=64 time=0.215 ms

But client code is not able to now make the connection successfully: root@myclient:/test# ./client 192.168.70.99 Error : Connect Failed : Connection refused

The client code is as follows:

include <sys/socket.h>

include <sys/types.h>

include <netinet/in.h>

include

include

include

include

include

include

include <arpa/inet.h>

include <sys/time.h>

include

void getlogtime(char *currentTime);

int main(int argc, char** argv) { int sockfd = 0; int bytesReceived = 0; char recvBuff[256]; memset(recvBuff, '0', sizeof(recvBuff)); struct sockaddr_in serv_addr;

if((sockfd = socket(AF_INET, SOCK_STREAM, 0))< 0)
{
    printf("\n Error : Could not create socket \n");
    return 1;
}

serv_addr.sin_family = AF_INET;
serv_addr.sin_port = htons(9600);
serv_addr.sin_addr.s_addr = inet_addr(argv[1]);

if(inet_pton(AF_INET, argv[1], &serv_addr.sin_addr)<=0)
{
    printf("\n inet_pton error occured\n");
    return 1;
}
if(connect(sockfd, (struct sockaddr *)&serv_addr, sizeof(serv_addr))<0)
{
    perror("Error : Connect Failed \n");
    return 1;
}

FILE *fp;
char fname[256];
char logtime[100];
int totalBytesRcvd = 0;

getlogtime(logtime);
sprintf(fname, "rcvdfle.%s", logtime);
fp = fopen(fname, "ab");
if(NULL == fp)
{
    printf("Error opening file");
    return 1;
}

getlogtime(logtime);
printf("Start time = {%s}\n", logtime);
while((bytesReceived = read(sockfd, recvBuff, 256)) > 0)
{
    totalBytesRcvd += bytesReceived;
    fwrite(recvBuff, 1,bytesReceived,fp);
    printf("%s \n", recvBuff);
}
printf("Total Bytes received %d\n",totalBytesRcvd);

getlogtime(logtime);
printf("End time = {%s}\n", logtime);

if(bytesReceived < 0)
{
    printf("\n Read Error \n");
}
close(sockfd);

return 0;

}

void getlogtime(char currentTime) { struct timeval curTime; int milli; time_t rawtime; struct tm timeinfo; char buffer [80];

gettimeofday(&curTime, NULL);
milli = curTime.tv_usec / 1000;
time(&rawtime);
timeinfo = localtime(&rawtime);
strftime(buffer, 80, "%H:%M:%S", timeinfo);
if(currentTime)
    sprintf(currentTime, "%s:%d", buffer, milli);
printf("current time: %s \n", currentTime);

}

What did I do wrong on my server side?

Thanks for the help!

[thehajime]

@speedingdaemon it looks like that you're configuring an IP address "192.168.70.99" to the host tap device and an interface of LKL server application. I believe it's not intentional, no ?

in that case, ping returns back from your host, not from LKL server application.

other things it's not clear to me is the following log lines

[ 0.000000] Kernel command line: ip=tcp virtio_mmio.device=268@0x1000000:1

this log should print the 2nd argument of lkl_start_kernel with some virtio_mmio strings. where was ip=tcp coming from ?

[ 1.420110] IP-Config: Guessing netmask 255.255.255.0
[ 1.420118] IP-Config: Complete:
[ 1.420121] device=eth0, hwaddr=ea:d2:db:db:5e:a3, ipaddr=206.0.0.0, mask=255.255.255.0, gw=255.255.255.255
[ 1.420139] host=206.0.0.0, domain=, nis-domain=(none)
[ 1.420141] bootserver=255.255.255.255, rootserver=255.255.255.255, rootpath=

I think the above lines are the result of ambiguous parameter of ip=tcp.

[speedingdaemon]

Sorry, didn't realized that I pasted the wrong snippet. Was trying out too many things in parallel. Don't have the host tap device anymore. Here is the correct code from server side:

include <sys/socket.h>

include <netinet/in.h>

include <arpa/inet.h>

include

include

include

include

include

include <sys/types.h>

include

include

int main(int argc, char* argv) { int listenfd = 0; int connfd = 0; struct sockaddr_in serv_addr; char sendBuff[1025]; int numrv; int totalbyteswritten = 0; int byteswritten = 0; struct lkl_netdev nd = NULL; int ret, nd_id = -1, nd_ifindex = -1;

nd = lkl_netdev_tap_create("lkl_ptt1", 0);
if (!nd) {
    fprintf(stderr, "init netdev failed\n");
    return -1;
}
ret = lkl_netdev_add(nd, NULL);
if (ret < 0) {
    fprintf(stderr, "failed to add netdev: %s\n",
            lkl_strerror(ret));
}
nd_id = ret;
ret = lkl_start_kernel(&lkl_host_ops, "mem=16M loglevel=8");
if (ret) {
    fprintf(stderr, "can't start kernel: %s\n", lkl_strerror(ret));
    return -1;
}

lkl_if_up(1);

if (nd_id >= 0) {
    nd_ifindex = lkl_netdev_get_ifindex(nd_id);
    if (nd_ifindex > 0)
        lkl_if_up(nd_ifindex);
    else
        fprintf(stderr, "failed to get ifindex for netdev id %d: %s\n",
                nd_id, lkl_strerror(nd_ifindex));
}

if (nd_ifindex >= 0) {
    unsigned int addr = inet_addr("192.168.70.99");
    int nmlen = atoi("22");

    if (addr != INADDR_NONE && nmlen > 0 && nmlen < 32) {
        ret = lkl_if_set_ipv4(nd_ifindex, addr, nmlen);
        if (ret < 0)
            fprintf(stderr, "failed to set IPv4 address: %s\n",
                    lkl_strerror(ret));
    }
}

if (nd_ifindex >= 0) {
    unsigned int addr = inet_addr("192.168.68.1");

    if (addr != INADDR_NONE) {
        ret = lkl_set_ipv4_gateway(addr);
        if (ret < 0)
            fprintf(stderr, "failed to set IPv4 gateway: %s\n",
                    lkl_strerror(ret));
    }
}
listenfd = lkl_sys_socket(AF_INET, SOCK_STREAM, 0);

memset(&serv_addr, '0', sizeof(serv_addr));
memset(sendBuff, '0', sizeof(sendBuff));

serv_addr.sin_family = AF_INET;
serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
serv_addr.sin_port = htons(9600);

while (1) {
    if (lkl_sys_bind(listenfd, (struct lkl_sockaddr*)&serv_addr, sizeof(serv_addr)) != 0 ) {
        printf("BIND: Failed to bind to port. Waiting....\n");
        sleep(10);
    } else {
        printf("BIND: Successful --- Serving Port:%d.\n", serv_addr.sin_port);
        break;
    }
}

if(lkl_sys_listen(listenfd, 10) == -1)
{
    printf("Failed to listen\n");
    return -1;
}
while(1)
{
    connfd = lkl_sys_accept(listenfd, (struct lkl_sockaddr*)NULL ,NULL);

    FILE *fp = fopen(argv[1],"rb");
    if(fp==NULL)
    {
        printf("File opern error");
        return 1;
    }

    while(1)
    {
        unsigned char buff[256]={0};
        int nread = fread(buff,1,256,fp);

        if(nread > 0)
        {
            byteswritten = lkl_sys_write(connfd, buff, nread);
            totalbyteswritten += byteswritten;
            printf("totalbyteswritten so far = {%d}\n", totalbyteswritten);
        }

        if (nread < 256)
        {
            if (feof(fp))
                printf("End of file\n");
            if (ferror(fp))
                printf("Error reading\n");
            break;
        }
    }
    printf("Total bytes sent = {%d}\n", totalbyteswritten);
    totalbyteswritten = 0;
    lkl_sys_close(connfd);
    sleep(1);
}
return 0;

}

and don't have the physical interface anymore: root@myserver:/lkl-test-client-server# ifconfig | grep lkl_ptt root@myserver:/lkl-test-client-server#

Here is the output from the lkl server process: ./server out [ 0.000000] Linux version 4.11.0+ (root@srv12) (gcc version 6.3.0 20170406 (Ubuntu 6.3.0-12ubuntu2) ) #3 Mon Jul 10 15:47:46 PDT 2017 [ 0.000000] bootmem address range: 0x7f137668b000 - 0x7f137768a000 [ 0.000000] On node 0 totalpages: 4095 [ 0.000000] free_area_init_node: node 0, pgdat 7f1379776160, node_mem_map 7f1376695e68 [ 0.000000] Normal zone: 56 pages used for memmap [ 0.000000] Normal zone: 0 pages reserved [ 0.000000] Normal zone: 4095 pages, LIFO batch:0 [ 0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768 [ 0.000000] pcpu-alloc: [0] 0 [ 0.000000] Built 1 zonelists in Zone order, mobility grouping off. Total pages: 4039 [ 0.000000] Kernel command line: mem=16M loglevel=8 virtio_mmio.device=268@0x1000000:1 [ 0.000000] PID hash table entries: 64 (order: -3, 512 bytes) [ 0.000000] Dentry cache hash table entries: 2048 (order: 2, 16384 bytes) [ 0.000000] Inode-cache hash table entries: 1024 (order: 1, 8192 bytes) [ 0.000000] Memory available: 16028k/0k RAM [ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] NR_IRQS:4096 [ 0.000000] lkl: irqs initialized [ 0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000001] lkl: time and timers initialized (irq2) [ 0.000015] pid_max: default: 4096 minimum: 301 [ 0.000045] Mount-cache hash table entries: 512 (order: 0, 4096 bytes) [ 0.000047] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes) [ 0.010658] console [lkl_console0] enabled [ 0.010693] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns [ 0.010703] xor: automatically using best checksumming function 8regs
[ 0.010852] NET: Registered protocol family 16 [ 0.182770] raid6: int64x1 gen() 2332 MB/s [ 0.354961] raid6: int64x1 xor() 1490 MB/s [ 0.527234] raid6: int64x2 gen() 2960 MB/s [ 0.699499] raid6: int64x2 xor() 2007 MB/s [ 0.871844] raid6: int64x4 gen() 2323 MB/s [ 1.044022] raid6: int64x4 xor() 2161 MB/s [ 1.216167] raid6: int64x8 gen() 3526 MB/s [ 1.388306] raid6: int64x8 xor() 2316 MB/s [ 1.388309] raid6: using algorithm int64x8 gen() 3526 MB/s [ 1.388311] raid6: .... xor() 2316 MB/s, rmw enabled [ 1.388313] raid6: using intx1 recovery algorithm [ 1.388555] clocksource: Switched to clocksource lkl [ 1.388669] NET: Registered protocol family 2 [ 1.388805] TCP established hash table entries: 512 (order: 0, 4096 bytes) [ 1.388809] TCP bind hash table entries: 512 (order: 0, 4096 bytes) [ 1.388814] TCP: Hash tables configured (established 512 bind 512) [ 1.388871] UDP hash table entries: 128 (order: 0, 4096 bytes) [ 1.388875] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes) [ 1.388964] virtio-mmio: Registering device virtio-mmio.0 at 0x1000000-0x100010b, IRQ 1. [ 1.389593] workingset: timestamp_bits=62 max_order=12 bucket_order=0 [ 1.390838] SGI XFS with ACLs, security attributes, no debug enabled [ 1.395261] io scheduler noop registered [ 1.395269] io scheduler deadline registered [ 1.395298] io scheduler cfq registered (default) [ 1.395301] io scheduler mq-deadline registered [ 1.395320] virtio-mmio virtio-mmio.0: Failed to enable 64-bit or 32-bit DMA. Trying to continue, but this might not work. [ 1.398525] NET: Registered protocol family 10 [ 1.399112] Segment Routing with IPv6 [ 1.399161] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver [ 1.399765] Btrfs loaded, crc32c=crc32c-generic [ 1.399880] Warning: unable to open an initial console. [ 1.399899] This architecture does not have kernel memory protection. BIND: Successful --- Serving Port:32805. write to fd netdev fails: Input/output error [ 2.578678] random: fast init done

[speedingdaemon]

@thehajime with me creating the tap on host and then in the server.c, i thought that is what net.sh and net-test.c were doing anyways. Is that not the case? Sorry, asking very basic question.

Referring to this: echo "== TAP (LKL net) tests ==" if [ -c /dev/net/tun ]; then sudo ip link set dev lkl_ptt1 down || true sudo ip tuntap del dev lkl_ptt1 mode tap || true sudo ip tuntap add dev lkl_ptt1 mode tap user $USER sudo ip link set dev lkl_ptt1 up sudo ip addr add dev lkl_ptt1 192.168.14.1/24

./net-test tap lkl_ptt1 192.168.14.1 192.168.14.2 24

sudo ip link set dev lkl_ptt1 down
sudo ip tuntap del dev lkl_ptt1 mode tap

fi

[thehajime]

@thehajime with me creating the tap on host and then in the server.c, i thought that is what net.sh and net-test.c were doing anyways.

yes, correct.

and don't have the physical interface anymore: root@myserver:/lkl-test-client-server# ifconfig | grep lkl_ptt root@myserver:/lkl-test-client-server#

so you need to have lkl_ptt1 on your host. what I mentioned was "do not assign the same IP address to the host and to the LKL app".

a host tap device is an endpoint and an LKL side is the other endpoint: those are connected by point-to-point fashion which should share the same IP network but should not assign the same IP address (as a general IP network requires).

./net-test tap lkl_ptt1 192.168.14.1 192.168.14.2 24

in this command of net.sh, argv[2] is the destination IP address, which is an IP address of the host tap device, and argv[3] is the IP address (and the mask at argv[4]) of the LKL side.

so you may want to assign an IP address something like 192.168.70.98.

Sorry, asking very basic question.

no problem. there is not document at all and would like to have a getting started document after this (hopefully).

[speedingdaemon]

I see. So I recreated the host lkl_ptt1 if: myserver# ifconfig lkl_ptt1 lkl_ptt1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 192.168.70.100 netmask 255.255.252.0 broadcast 0.0.0.0 inet6 fe80::b85a:bff:fe9c:17b8 prefixlen 64 scopeid 0x20 ether ba:5a:0b:9c:17:b8 txqueuelen 1000 (Ethernet) RX packets 18 bytes 1452 (1.4 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 31 bytes 2402 (2.4 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

sudo ip tuntap del dev lkl_ptt1 mode tap sudo ip tuntap add dev lkl_ptt1 mode tap user root sudo ip link set dev lkl_ptt1 up sudo ip addr add dev lkl_ptt1 sudo ip addr add dev lkl_ptt1 192.168.70.100/22

Assigned it .100/22 IP.

Still client is not able to connect.

Server-side log is: ./server out [ 0.000000] Linux version 4.11.0+ (root@srv12) (gcc version 6.3.0 20170406 (Ubuntu 6.3.0-12ubuntu2) ) #3 Mon Jul 10 15:47:46 PDT 2017 [ 0.000000] bootmem address range: 0x7f31625da000 - 0x7f31635d9000 [ 0.000000] On node 0 totalpages: 4095 [ 0.000000] free_area_init_node: node 0, pgdat 7f31656c5160, node_mem_map 7f31625e27b0 [ 0.000000] Normal zone: 56 pages used for memmap [ 0.000000] Normal zone: 0 pages reserved [ 0.000000] Normal zone: 4095 pages, LIFO batch:0 [ 0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768 [ 0.000000] pcpu-alloc: [0] 0 [ 0.000000] Built 1 zonelists in Zone order, mobility grouping off. Total pages: 4039 [ 0.000000] Kernel command line: mem=16M loglevel=8 virtio_mmio.device=268@0x1000000:1 [ 0.000000] PID hash table entries: 64 (order: -3, 512 bytes) [ 0.000000] Dentry cache hash table entries: 2048 (order: 2, 16384 bytes) [ 0.000000] Inode-cache hash table entries: 1024 (order: 1, 8192 bytes) [ 0.000000] Memory available: 16028k/0k RAM [ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] NR_IRQS:4096 [ 0.000000] lkl: irqs initialized [ 0.000000] clocksource: lkl: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000001] lkl: time and timers initialized (irq2) [ 0.000022] pid_max: default: 4096 minimum: 301 [ 0.000057] Mount-cache hash table entries: 512 (order: 0, 4096 bytes) [ 0.000060] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes) [ 0.009751] console [lkl_console0] enabled [ 0.009786] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns [ 0.009802] xor: automatically using best checksumming function 8regs
[ 0.010824] NET: Registered protocol family 16 [ 0.183792] raid6: int64x1 gen() 2327 MB/s [ 0.355899] raid6: int64x1 xor() 1483 MB/s [ 0.527987] raid6: int64x2 gen() 2956 MB/s [ 0.700118] raid6: int64x2 xor() 2332 MB/s [ 0.872354] raid6: int64x4 gen() 3228 MB/s [ 1.044782] raid6: int64x4 xor() 2490 MB/s [ 1.216985] raid6: int64x8 gen() 3539 MB/s [ 1.389050] raid6: int64x8 xor() 2310 MB/s [ 1.389053] raid6: using algorithm int64x8 gen() 3539 MB/s [ 1.389055] raid6: .... xor() 2310 MB/s, rmw enabled [ 1.389057] raid6: using intx1 recovery algorithm [ 1.389451] clocksource: Switched to clocksource lkl [ 1.389640] NET: Registered protocol family 2 [ 1.389818] TCP established hash table entries: 512 (order: 0, 4096 bytes) [ 1.389823] TCP bind hash table entries: 512 (order: 0, 4096 bytes) [ 1.389829] TCP: Hash tables configured (established 512 bind 512) [ 1.389886] UDP hash table entries: 128 (order: 0, 4096 bytes) [ 1.389891] UDP-Lite hash table entries: 128 (order: 0, 4096 bytes) [ 1.389990] virtio-mmio: Registering device virtio-mmio.0 at 0x1000000-0x100010b, IRQ 1. [ 1.390761] workingset: timestamp_bits=62 max_order=12 bucket_order=0 [ 1.392060] SGI XFS with ACLs, security attributes, no debug enabled [ 1.396680] io scheduler noop registered [ 1.396688] io scheduler deadline registered [ 1.396733] io scheduler cfq registered (default) [ 1.396751] io scheduler mq-deadline registered [ 1.396788] virtio-mmio virtio-mmio.0: Failed to enable 64-bit or 32-bit DMA. Trying to continue, but this might not work. [ 1.399937] NET: Registered protocol family 10 [ 1.400605] Segment Routing with IPv6 [ 1.400639] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver [ 1.401273] Btrfs loaded, crc32c=crc32c-generic [ 1.401368] Warning: unable to open an initial console. [ 1.401385] This architecture does not have kernel memory protection. BIND: Successful --- Serving Port:32805. [ 2.539563] random: fast init done

Client log is: client# ./client 192.168.70.99 Error : Connect Failed : No route to host root@srv12:~/users/ar

This is what my other client-side machine is reporting: myclient$ ping 192.168.70.100 PING 192.168.70.100 (192.168.70.100) 56(84) bytes of data. 64 bytes from 192.168.70.100: icmp_seq=1 ttl=64 time=0.205 ms 64 bytes from 192.168.70.100: icmp_seq=2 ttl=64 time=0.258 ms 64 bytes from 192.168.70.100: icmp_seq=3 ttl=64 time=0.252 ms ^C --- 192.168.70.100 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2038ms rtt min/avg/max/mdev = 0.205/0.238/0.258/0.026 ms mysclient$ ping 192.168.70.99 PING 192.168.70.99 (192.168.70.99) 56(84) bytes of data. From 192.168.70.20 icmp_seq=1 Destination Host Unreachable ^C --- 192.168.70.99 ping statistics --- 6 packets transmitted, 0 received, +1 errors, 100% packet loss, time 5114ms pipe 3

Server-side tcpdump is showing this: myserver# tcpdump host 192.168.70.20 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes 18:13:19.753526 IP 192.168.70.20 > srv12: ICMP echo request, id 19137, seq 1, length 64 18:13:19.753554 IP srv12 > 192.168.70.20: ICMP echo reply, id 19137, seq 1, length 64 18:13:20.768467 IP 192.168.70.20 > srv12: ICMP echo request, id 19137, seq 2, length 64 18:13:20.768495 IP srv12 > 192.168.70.20: ICMP echo reply, id 19137, seq 2, length 64 18:13:21.792445 IP 192.168.70.20 > srv12: ICMP echo request, id 19137, seq 3, length 64 18:13:21.792473 IP srv12 > 192.168.70.20: ICMP echo reply, id 19137, seq 3, length 64 18:13:23.590283 ARP, Request who-has 192.168.70.99 tell 192.168.70.20, length 46 18:13:24.608445 ARP, Request who-has 192.168.70.99 tell 192.168.70.20, length 46 18:13:25.632402 ARP, Request who-has 192.168.70.99 tell 192.168.70.20, length 46 18:13:26.656480 ARP, Request who-has 192.168.70.99 tell 192.168.70.20, length 46 18:13:27.680399 ARP, Request who-has 192.168.70.99 tell 192.168.70.20, length 46 18:13:28.704435 ARP, Request who-has 192.168.70.99 tell 192.168.70.20, length 46

Am I correct when I think that ping to the application tap interface anyways will not work for me if my server is not responding to the icmp requests..?

[thehajime]

I found you're using two separate machines for the client and server. then you probably need to configure a bridge device, as https://github.com/lkl/linux/issues/133#issuecomment-200260776 does.

with a single host communication via tap (like net.sh does), you dont need a bridge device.

[speedingdaemon]

Thanks. that works.

myserver# brctl show bridge name bridge id STP enabled interfaces testbridge 8000.000743311490 no ens15f4 lkl_ptt1

myserver# ifconfig lkl_ptt1 lkl_ptt1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.75.88 netmask 255.255.255.0 broadcast 0.0.0.0 inet6 fe80::a44c:6dff:fe84:e6e4 prefixlen 64 scopeid 0x20 ether a6:4c:6d:84:e6:e4 txqueuelen 1000 (Ethernet) RX packets 221 bytes 20260 (20.2 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 344 bytes 26674 (26.6 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

myserver# ifconfig ens15f4 ens15f4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500 inet 192.168.75.101 netmask 255.255.255.0 broadcast 0.0.0.0 inet6 fe80::207:43ff:fe31:1490 prefixlen 64 scopeid 0x20 ether 00:07:43:31:14:90 txqueuelen 1000 (Ethernet) RX packets 2311906 bytes 253362099 (253.3 MB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 316194250 bytes 20788272053 (20.7 GB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 37

myclient# ifconfig ens15f4 ens15f4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.75.100 netmask 255.255.255.0 broadcast 0.0.0.0 inet6 fe80::207:43ff:fe31:cd0 prefixlen 64 scopeid 0x20 ether 00:07:43:31:0c:d0 txqueuelen 1000 (Ethernet) RX packets 2554463 bytes 714350817 (714.3 MB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 2291616 bytes 246279570 (246.2 MB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 35

server.c code change: if (nd_ifindex >= 0) { unsigned int addr = inet_addr("192.168.75.99"); int nmlen = atoi("24");

    if (addr != INADDR_NONE && nmlen > 0 && nmlen < 32) {
        ret = lkl_if_set_ipv4(nd_ifindex, addr, nmlen);
        if (ret < 0)
            fprintf(stderr, "failed to set IPv4 address: %s\n",
                    lkl_strerror(ret));
    }
}

With the above changes, I can see my client is able to fetch the file from the server.

But there is still one more issue that I seem to be having. My client doesn't work if I run it right after bringing up my server-lkl process. Looks like ping from the server, where my client is running, takes a while to go thru until when client is not able to connect to the server side.

Is there any way I can speed up the networking stack on server/client to work faster so that I dont have to "guess" how long I need to wait before my client can connect to the server?

[thehajime]

But there is still one more issue that I seem to be having. My client doesn't work if I run it right after bringing up my server-lkl process. Looks like ping from the server, where my client is running, takes a while to go thru until when client is not able to connect to the server side.

Is there any way I can speed up the networking stack on server/client to work faster so that I dont have to "guess" how long I need to wait before my client can connect to the server?

though the solution may depend on how the exact situation is, a common tip is to use static MAC address for the network device on LKL app, since it is always changing in every execution (of an LKL app) and may prevent the communication while arp cache generated by the previous execution of LKL app is remained on the sender side (and once it's expired the communication may start successfully).

You can use LKL_HIJACK_NET_MAC= for a static MAC address.

[speedingdaemon]

Ok. This works. char mac_str[] = "00:80:C8:E3:4C:BD"; struct lkl_netdev_args nd_args;

nd = lkl_netdev_tap_create("lkl_ptt1", 0);

if (!nd) {
    fprintf(stderr, "init netdev failed\n");
    return -1;
}
memset(&nd_args, 0, sizeof(struct lkl_netdev_args));
__lkl__u8 mac[LKL_ETH_ALEN] = {0};
ret = parse_mac_str(mac_str, mac);
if (ret < 0) {
    fprintf(stderr, "failed to parse mac\n");
    return -1;
} else if (ret > 0) {
    nd_args.mac = mac;
} else {
    nd_args.mac = NULL;
}
ret = lkl_netdev_add(nd, &nd_args);

By the way, do I always have to create a net_device for the virtio net backend to have my server be able to get packets? Wondering if what I have in my hello-world code is not throwaway.

[speedingdaemon]

One more question. How do I debug/figure out that what I have on my server side has actually bypassed the host kernel network stack? Any tools/logs/etc?

[tavip]

You can try reading /proc/net/tcp from LKL.

[thehajime]

@speedingdaemon good to see you managed to do it.

By the way, do I always have to create a net_device for the virtio net backend to have my server be able to get packets? Wondering if what I have in my hello-world code is not throwaway.

I'm afraid that I understand your point, but try to answer anyway.

For the tap/macvtap backend, yes you need to have a correspond host one. For the raw socket backend, you can specify a net device on the host (e.g., eth0). For the DPDK/vde backend, you need to specify an identifier for each backend (but not a net device of Linux host) .

[speedingdaemon]

You can try reading /proc/net/tcp from LKL.

Sorry, what do you mean by "from LKL"? How do I do that?

[tavip]

Sorry, what do you mean by "from LKL"? How do I do that?

Use lkl_sys_open on "/proc/net/tcp" and lkl_sys_read to read its content and print it somewhere. Perhaps we should add some helpers in lib/utils.c for that...

[speedingdaemon]

I see. In this document, https://netdevconf.org/1.2/papers/jerry_chu.pdf, Jerry mentioned that they added a CLI interface to the LKL process. How can I use that? Did he commit it to mainline?

[thehajime]

I see. In this document, https://netdevconf.org/1.2/papers/jerry_chu.pdf, Jerry mentioned that they added a CLI interface to the LKL process. How can I use that? Did he commit it to mainline?

Yes.

https://github.com/lkl/linux/pull/162

Hit Ctrl-Z to enter the debug console, which is great facility actually. Do not forget to add LKL_HIJACK_DEBUG=0x100 for your env variable.

(cc: @liuyuan10, @hkchu)

[speedingdaemon]

Doesn't work for me. this is what happens when I try: ^Z [1]+ Stopped ./run.sh -bash-4.2$

run.sh is my shell script to launch the process

[hkchu]

On Wed, Sep 27, 2017 at 5:26 PM, speedingdaemon notifications@github.com wrote:

Doesn't work for me. this is what happens when I try:

It looks like your control key got interrupted by run.sh first. If you run LKL directly (outside of a shell script) you shouldn't have this problem (i think).

^Z

[1]+ Stopped ./run.sh -bash-4.2$

run.sh is my shell script to launch the process

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lkl/linux/issues/355#issuecomment-332692500, or mute the thread https://github.com/notifications/unsubscribe-auth/AR3_fWIObB-AMcr46SzHVRyiI-hw9QOfks5smueogaJpZM4OTlR9 .

[speedingdaemon]

Tried that also. Running into the same issue.