iqiyi / dpvs

DPVS is a high performance Layer-4 load balancer based on DPDK.
Other
2.93k stars 713 forks source link

keepalived crashing on SMTP send #388

Open Tolyar opened 5 years ago

Tolyar commented 5 years ago

Hi. I've checked DPVS with small configuration file and all was fine until I pushed configuration with a lot of VS sections keepalived was crashed. The same configuration works fine on vanilla IPVS.

There is bt:

(gdb) thread apply all bt

Thread 1 (Thread 0x7f2e070c0840 (LWP 1028756)):

0 0x00007f2e05e05207 in raise () from /lib64/libc.so.6

1 0x00007f2e05e068f8 in abort () from /lib64/libc.so.6

2 0x00007f2e05dfe026 in __assert_fail_base () from /lib64/libc.so.6

3 0x00007f2e05dfe0d2 in __assert_fail () from /lib64/libc.so.6

4 0x00000000004254a5 in thread_fetch (m=0x1990960, fetch=fetch@entry=0x7ffecd2ea930) at scheduler.c:608

5 0x00000000004255fc in launch_scheduler () at scheduler.c:776

6 0x0000000000406b1b in start_check_child () at check_daemon.c:318

7 0x00000000004030c1 in start_keepalived () at main.c:80

8 main (argc=, argv=) at main.c:303

(gdb) p thread $1 = {id = 4505, type = 11 '\v', next = 0x0, prev = 0x0, master = 0x1990960, func = 0x403b70 , arg = 0x1b70f40, sands = {tv_sec = 1550071023, tv_usec = 627754}, u = {val = 24, fd = 24, c = {pid = 24, status = 0}}}

This is tail for keepalived output

Adding service [10.10.0.109]:80 to VS [10.10.20.75]:80 [sockopt_msg_recv] errcode set in socket msg#205 header: already exist(-3) IPVS: Operation now in progress Remote SMTP server [0.0.0.0]:25 connected. SMTP alert successfully sent. keepalived.unstripped: scheduler.c:608: thread_fetch: Assertion `0' failed. Healthcheck child process(1033276) died: Exiting Stopping Keepalived v1.2.13 (01/15,2019)

Problem happened not on first email but after tens emails. Disabling of email sending helps. Looks like this is bug in original keepalived sources (I've found reports on keepalived bug tracker). Do you plan to porting fresh keepalived to DPVS? Looks like a lot of bugs in keepalived was fixed in 2.x branch.

ywc689 commented 5 years ago

This keepalived does have many bugs, we are considering to improve it.