Reliable crashes on HTTP test on Ubuntu 12.04

[jakubd]

Centinel reliably crashes during baseline testing on Ubuntu 12.04 on the latter end of the HTTP test:

2015-12-09 22:26:42,817 client.py(line 140) INFO: Centinel started.
2015-12-09 22:26:44,957 client.py(line 290) INFO: tcpdump started...
2015-12-09 22:26:46,959 baseline.py(line 57) INFO: Testing input file world.csv...
2015-12-09 22:26:46,967 baseline.py(line 191) INFO: Running HTTP GET requests...
*** longjmp causes uninitialized stack frame ***: /usr/bin/python terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f3c62dace57]
/lib/x86_64-linux-gnu/libc.so.6(+0x109dcd)[0x7f3c62dacdcd]
/lib/x86_64-linux-gnu/libc.so.6(__longjmp_chk+0x33)[0x7f3c62dacd33]
/usr/lib/x86_64-linux-gnu/libcurl.so.4(+0xe025)[0x7f3c5feea025]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f3c63fc3cb0]
/lib/x86_64-linux-gnu/libc.so.6(__select+0x33)[0x7f3c62d8f6f3]
/usr/bin/python[0x51ea23]
/usr/bin/python(PyEval_EvalFrameEx+0x2a5)[0x4f95a5]
/usr/bin/python(PyEval_EvalCodeEx+0x1a2)[0x5008c2]
/usr/bin/python(PyEval_EvalFrameEx+0x7b8)[0x4f9ab8]
/usr/bin/python(PyEval_EvalCodeEx+0x1a2)[0x5008c2]
/usr/bin/python(PyEval_EvalFrameEx+0x7b8)[0x4f9ab8]
/usr/bin/python(PyEval_EvalCodeEx+0x1a2)[0x5008c2]
/usr/bin/python(PyEval_EvalFrameEx+0x7b8)[0x4f9ab8]
/usr/bin/python(PyEval_EvalFrameEx+0xa02)[0x4f9d02]
/usr/bin/python(PyEval_EvalFrameEx+0xa02)[0x4f9d02]
/usr/bin/python(PyEval_EvalCodeEx+0x1a2)[0x5008c2]
/usr/bin/python(PyEval_EvalFrameEx+0x7b8)[0x4f9ab8]
/usr/bin/python(PyEval_EvalCodeEx+0x1a2)[0x5008c2]
/usr/bin/python(PyEval_EvalFrameEx+0x7b8)[0x4f9ab8]
/usr/bin/python(PyEval_EvalFrameEx+0xa02)[0x4f9d02]
/usr/bin/python(PyEval_EvalCodeEx+0x1a2)[0x5008c2]
/usr/bin/python(PyRun_FileExFlags+0xdd)[0x4990cd]
/usr/bin/python(PyRun_SimpleFileExFlags+0x216)[0x499736]
/usr/bin/python(Py_Main+0x87f)[0x49a9cf]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f3c62cc476d]
/usr/bin/python[0x41bb31]
======= Memory map: ========
00400000-00671000 r-xp 00000000 b6:cf3b1 526151                          /usr/bin/python2.7
00871000-00872000 r--p 00271000 b6:cf3b1 526151                          /usr/bin/python2.7
00872000-008db000 rw-p 00272000 b6:cf3b1 526151                          /usr/bin/python2.7
008db000-008ed000 rw-p 00000000 00:00 0
015a0000-02322000 rw-p 00000000 00:00 0                                  [heap]
7f3c2b9f9000-7f3c2b9fa000 ---p 00000000 00:00 0
7f3c2b9fa000-7f3c2c3fa000 rw-p 00000000 00:00 0
7f3c2ebfe000-7f3c2ebff000 ---p 00000000 00:00 0
7f3c2ebff000-7f3c2f5ff000 rw-p 00000000 00:00 0
7f3c30000000-7f3c3015f000 rw-p 00000000 00:00 0
7f3c3015f000-7f3c34000000 ---p 00000000 00:00 0
7f3c36bfe000-7f3c36bff000 ---p 00000000 00:00 0
7f3c36bff000-7f3c375ff000 rw-p 00000000 00:00 0
7f3c38000000-7f3c3827d000 rw-p 00000000 00:00 0
7f3c3827d000-7f3c3c000000 ---p 00000000 00:00 0
7f3c3c000000-7f3c3c113000 rw-p 00000000 00:00 0
7f3c3c113000-7f3c40000000 ---p 00000000 00:00 0
7f3c402be000-7f3c402eb000 rw-p 00000000 00:00 0
7f3c42bfe000-7f3c42bff000 ---p 00000000 00:00 0
7f3c42bff000-7f3c435ff000 rw-p 00000000 00:00 0
7f3c435ff000-7f3c43600000 ---p 00000000 00:00 0
7f3c43600000-7f3c44000000 rw-p 00000000 00:00 0
7f3c44000000-7f3c4403a000 rw-p 00000000 00:00 0
7f3c4403a000-7f3c48000000 ---p 00000000 00:00 0
7f3c48000000-7f3c48084000 rw-p 00000000 00:00 0
7f3c48084000-7f3c4c000000 ---p 00000000 00:00 0
7f3c4c000000-7f3c4c6ae000 rw-p 00000000 00:00 0
7f3c4c6ae000-7f3c50000000 ---p 00000000 00:00 0
7f3c50000000-7f3c501c9000 rw-p 00000000 00:00 0
7f3c501c9000-7f3c54000000 ---p 00000000 00:00 0
7f3c54000000-7f3c5409b000 rw-p 00000000 00:00 0
7f3c5409b000-7f3c58000000 ---p 00000000 00:00 0
7f3c5802c000-7f3c5802d000 ---p 00000000 00:00 0
7f3c5802d000-7f3c58a2d000 rw-p 00000000 00:00 0
7f3c5a830000-7f3c5a831000 ---p 00000000 00:00 0
7f3c5a831000-7f3c5b231000 rw-p 00000000 00:00 0
7f3c5b231000-7f3c5b232000 ---p 00000000 00:00 0
7f3c5b232000-7f3c5bc32000 rw-p 00000000 00:00 0
7f3c5bc32000-7f3c5bc39000 r-xp 00000000 b6:cf3b1 1048901                 /lib/x86_64-linux-gnu/libnss_dns-2.15.so
7f3c5bc39000-7f3c5be38000 ---p 00007000 b6:cf3b1 1048901                 /lib/x86_64-linux-gnu/libnss_dns-2.15.so
7f3c5be38000-7f3c5be39000 r--p 00006000 b6:cf3b1 1048901                 /lib/x86_64-linux-gnu/libnss_dns-2.15.so
7f3c5be39000-7f3c5be3a000 rw-p 00007000 b6:cf3b1 1048901                 /lib/x86_64-linux-gnu/libnss_dns-2.15.so
7f3c5be3a000-7f3c5be43000 r-xp 00000000 b6:cf3b1 1048898                 /lib/x86_64-linux-gnu/libcrypt-2.15.so
7f3c5be43000-7f3c5c043000 ---p 00009000 b6:cf3b1 1048898                 /lib/x86_64-linux-gnu/libcrypt-2.15.so
7f3c5c043000-7f3c5c044000 r--p 00009000 b6:cf3b1 1048898                 /lib/x86_64-linux-gnu/libcrypt-2.15.so
7f3c5c044000-7f3c5c045000 rw-p 0000a000 b6:cf3b1 1048898                 /lib/x86_64-linux-gnu/libcrypt-2.15.so
7f3c5c045000-7f3c5c073000 rw-p 00000000 00:00 0
7f3c5c073000-7f3c5c111000 r-xp 00000000 b6:cf3b1 524574                  /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f3c5c111000-7f3c5c311000 ---p 0009e000 b6:cf3b1 524574                  /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f3c5c311000-7f3c5c313000 r--p 0009e000 b6:cf3b1 524574                  /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f3c5c313000-7f3c5c315000 rw-p 000a0000 b6:cf3b1 524574                  /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f3c5c315000-7f3c5c316000 rw-p 00000000 00:00 0
7f3c5c316000-7f3c5c35b000 r-xp 00000000 b6:cf3b1 525425                  /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f3c5c35b000-7f3c5c55b000 ---p 00045000 b6:cf3b1 525425                  /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f3c5c55b000-7f3c5c55d000 r--p 00045000 b6:cf3b1 525425                  /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f3c5c55d000-7f3c5c55f000 rw-p 00047000 b6:cf3b1 525425                  /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f3c5c55f000-7f3c5c560000 rw-p 00000000 00:00 0
7f3c5c560000-7f3c5c56e000 r-xp 00000000 b6:cf3b1 525477                  /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f3c5c56e000-7f3c5c76d000 ---p 0000e000 b6:cf3b1 525477                  /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f3c5c76d000-7f3c5c76e000 r--p 0000d000 b6:cf3b1 525477                  /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f3c5c76e000-7f3c5c76f000 rw-p 0000e000 b6:cf3b1 525477                  /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f3c5c76f000-7f3c5c797000 r-xp 00000000 b6:cf3b1 525494                  /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f3c5c797000-7f3c5c996000 ---p 00028000 b6:cf3b1 525494                  /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f3c5c996000-7f3c5c997000 r--p 00027000 b6:cf3b1 525494                  /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f3c5c997000-7f3c5c998000 rw-p 00028000 b6:cf3b1 525494                  /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f3c5c998000-7f3c5c99b000 r-xp 00000000 b6:cf3b1 1048796                 /lib/x86_64-linux-gnu/libkeyutils.so.1.4
7f3c5c99b000-7f3c5cb9a000 ---p 00003000 b6:cf3b1 1048796                 /lib/x86_64-linux-gnu/libkeyutils.so.1.4
7f3c5cb9a000-7f3c5cb9b000 r--p 00002000 b6:cf3b1 1048796                 /lib/x86_64-linux-gnu/libkeyutils.so.1.4

Stackflow hive wisdom indicates this might be a curl bug with an easy fix relating to how signals are processed in pycurl: https://stackoverflow.com/questions/9191668/error-longjmp-causes-uninitialized-stack-frame

I tested this solution on a system running Ubuntu 12.04 and it seems to fix the issue PR incoming.

[rpanah]

@jakubd, has this been addressed?

[sneft]

Coincidentally we just encountered this yesterday on a new install running 0.1.5.4.1. (The fix is in the new package which hasn't been released yet) We made the fix manually and it solved the problem.

[rpanah]

The fixes you guys proposed have been merged. Can you try with the latest git version to confirm that the latest version has the fix? I want to release a new package (1.5.4.2) that will include a couple of updates (including this).

[rpanah]

Looks like this is caused by nodes running out-dated software (older versions of curl). @sneft, I think we can close this since there's not much can be done in Centinel to stop this from happening.

[sahilgupta221]

Hi everyone, Is there any documentation on how to use this centinel program? How can we write our own custom program and make it work with ICLab nodes?

[arianniaki]

Hi Sahil, You can find more information in the Wiki section of the repository and the README file. https://github.com/iclab/centinel/wiki/How-to-write-an-experiment-and-run-it-in-Centinel

Please email us for more questions.

[sahilgupta221]

@arianniaki Thanks! I will look into it. :)