Does libuv work out of the box with LKL?

[pmembrey]

Hi guys,

I'm playing with a small network test app based on libuv and trying to get it to work with lkl. I suspect I'm just doing something stupid and I'd be very happy to write up some docs afterwards to help others who might get stuck in a similar situation.

I can replicate the issue with this libuv example app (tcp-echo-server):

https://github.com/delaemon/libuv-tutorial/blob/master/tcp-echo-server.c

The app triggers a SIGABRT in the same location as my app.

From what I can tell, the TCP socket I try to open (literally just binding and listening) gets a really high file descriptor (512). I'm not sure if that's normal for an lkl app but when run normally, that socket gets 10. Running lsof on the process after gdb has grabbed it does not show any open sockets - but I'm assuming that's because it is simulated in lkl itself.

I can confirm from gdb that 512 is what is present straight after calling uv_tcp_bind.

The abort seems to be occuring at linux-core.c:245:

https://github.com/libuv/libuv/blob/v1.x/src/unix/linux-core.c#L245

I've tried this with the latest stable release of libuv (v1.20.1) and built lkl from master.

Basically, I'm not sure if it's something I've set up wrong, something I need to configure but haven't, or whether I should even expect libuv to work (although from previous issues, it looks like it might have been working previously).

Any help gratefully received :)

[thehajime]

@pmembrey libuv should work as node.js application can be hijacked (#369).

from a glance, the problem looks like a regression of the commit https://github.com/lkl/linux/commit/8f6b03d865d0fac79e9d9e011196e62df070bdce, which tries to allow mixture of host's fd and lkl's fd in epoll_wait(2).

the number of file descriptor starts from FD_SETSIZE/2 (maybe 512) if an application opens it from LKL kernel (https://github.com/lkl/linux/blob/master/tools/lkl/lib/hijack/init.c#L515) while the host fds are kept the value as the host kernel returns. in this way LKL hijack lib can distinguish which kernel we should ask (with select/poll/epoll_wait etc).

so first recommendation would be reverting the above commit to see if your application works fine or not.

[pmembrey]

@thehajime thanks for the super quick reply!

I've tried reverting to https://github.com/lkl/linux/commit/8f6b03d865d0fac79e9d9e011196e62df070bdce and also to 9d5deaeaac4b7437d6c39ac2593368e0b052f73f but have not been able to get that demo application to work.

Would it be possible for you to test the demo app? If it works for you, then it's clearly something I'm doing wrong my end :)

[thehajime]

I could reproduce exactly what you explained on my local machine. during a gdb session,

299     int uv__epoll_ctl(int epfd, int op, int fd, struct uv__epoll_event* events) {
300     #if defined(__NR_epoll_ctl)
301       return syscall(__NR_epoll_ctl, epfd, op, fd, events);
302     #else
303       return errno = ENOSYS, -1;
304     #endif
305     }
(gdb) 

epoll_ctl(2) is directly called without libc, which makes hijack lib impossible to intercept because the interception is done by LD_PRELOAD, not the other way like ptrace.

This wonders and reminds me how others (like @octaviansoldea for his node.js integration, https://software.intel.com/en-us/articles/minimize-nodejs-io-bottlenecks) uses libuv in their scenario. and eventually found the following conversation.

https://github.com/octaviansoldea/node/commit/03377894c04b63e1b48f5df517a91873bb0b0365#commitcomment-23128342

in summary, we can't use the stock version of libuv, have to modify src/unix/linux-syscalls.c.

[pmembrey]

That looks like quite a project in itself :grinning:

I was hoping I'd find something like libuv-lkl but unfortunately I didn't come across anything.

[octaviansoldea]

Thank you for your feedback. Please consider using

https://github.com/octaviansoldea/node/tree/lkl_dpdk

The last commit of this repository includes modifications of libuv, as a part of Node.js.

[pmembrey]

Hi @octaviansoldea, thanks for the link! I checked out the project but I'm not actually using Node.js just libuv. So I was hoping for something cleaner. I'll take a look at the libuv code and see if it'd a relatively easy change.

I've not used dpdk before but a number of my machines do have Intel cards, so I'll definitely take a look.

Any objection to me looking at your adaptations for inspiration? :)

[octaviansoldea]

Hi @pmembrey

While I cannot distribute the code, I will be happy to try to help you install the code in my free time. Please let me know if you encounter any difficulties.