vfs_lxip stucks reliable on all hw platforms

[skalk]

At least the nightly tests regarding netperf with lxip and test-libc_connect_lxip reliably stuck across all different hardware on the hw kernel. I've investigated that in the test-libc_connect_lxip the client successfully sends a packet through the nic_router to the server, but the server application never receives it, because it does not wait for signals. When instrumenting the kernel, I could trace that the server once wait for a signal and receives one, but never again calls await_signal syscall. Therefore, it obviously cannot receive a signal not packet. Next I try to investigate the state of the server component, and why it never again calls the block_io_signal function. I expect a deadlock in the libc/vfs_lxip model. Due to a quite different runtime behaviour on hw (no global signal receiving thread), it would explain the occurrence only on this kernel. Other suggestions are welcome.

It is quite probable that issue #3160 is related to this one.

[nfeske]

Thanks for sharing your intermediate analysis.

[skalk]

Ok, it is no deadlock, but a lifelock. The server component's ep never blocks again from a certain point on. When sampling the ip of the server component ep thread, I get the following sample:

            0x4b970            0x4b970  ld.lib.so                       Genode::Avl_node_base::remove(Genode::Avl_node_base::Policy&)(T) +    0x0 = 0x4b970
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/avl_tree.cc:121
            0xd27a0            0xd27a0  ld.lib.so                       __gxx_personality_v0(T) +  0x186 = 0xd261a
            0xcb1cb            0xcb1cb  libc.lib.so                     getopt_internal(t) +  0x13b = 0xcb090
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/getopt_long.c:406
            0xcb1e1            0xcb1e1  libc.lib.so                     getopt_internal(t) +  0x151 = 0xcb090
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/getopt_long.c:424
            0x49950            0x49950  ld.lib.so                       Genode::Allocator_avl_base::_alloc_block_metadata()(T) +    0x0 = 0x49950
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/allocator_avl.cc:99
            0x830cd            0x830cd  ld.lib.so                       Genode::Heap::free(void*, unsigned long)(T) +   0x8d = 0x83040
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/heap.cc:243
            0xcb1a1            0xcb1a1  libc.lib.so                     getopt_internal(t) +  0x111 = 0xcb090
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/getopt_long.c:430
            0xa5c80            0xa5c80  ld.lib.so                       Genode::Slab::Block::_slab_entry(int)(T) +   0x20 = 0xa5c60
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/misc_math.h:54
            0xd226e            0xd226e  ld.lib.so                       read_encoded_value(_Unwind_Context*, unsigned char, unsigned char const*, unsigned long*)(t) +   0x41 = 0xd222d
            0xd282a            0xd282a  libc.lib.so                     strtoul(T) +    0xa = 0xd2820
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/strtoul.c:49
            0xcab60            0xcab60  libc.lib.so                     parse_long_options(t) +    0x0 = 0xcab60
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/getopt_long.c:201
            0x49ac0            0x49ac0  libc.lib.so                     pthread_getspecific_int(t) +    0x0 = 0x49ac0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:224
            0xd1d4f            0xd1d4f  libc.lib.so                     strtoimax(T) +   0x2f = 0xd1d20
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/strtoimax.c:49
            0xa5c80            0xa5c80  ld.lib.so                       Genode::Slab::Block::_slab_entry(int)(T) +   0x20 = 0xa5c60
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/misc_math.h:54
            0x4af48            0x4af48  libc.lib.so                     pthread_attr_setstackaddr_int(t) +   0x58 = 0x4aef0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:260
            0xccc70            0xccc70  ld.lib.so                       _Unwind_GetRegionStart(t) +    0x0 = 0xccc70
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:373
            0xd362c            0xd362c  ld.lib.so                       std::type_info::__is_pointer_p() const(t) +    0x0 = 0xd362c
            0x49c8d            0x49c8d  libc.lib.so                     pthread_mutex_destroy_int(t) +   0x3d = 0x49c50
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:228
            0xcd530            0xcd530  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0xcadb0            0xcadb0  ld.lib.so                       execute_cfa_program(t) +   0xc0 = 0xcacf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:953
            0xd266a            0xd266a  ld.lib.so                       __gxx_personality_v0(T) +   0x50 = 0xd261a
            0xce691            0xce691  libc.lib.so                     r_sort_b(t) +    0x1 = 0xce690
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/radixsort.c:241
            0xa5c88            0xa5c88  ld.lib.so                       Genode::Slab::Block::_slab_entry(int)(T) +   0x28 = 0xa5c60
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/misc_math.h:55
            0x84493            0x84493  ld.lib.so                       Genode::Heap::alloc(unsigned long, void**)(T) +   0xa3 = 0x843f0
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/heap.cc:229
            0xcebf9            0xcebf9  ld.lib.so                       _Unwind_Find_FDE(t) +   0x39 = 0xcebc0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:1014
            0xcebf9            0xcebf9  ld.lib.so                       _Unwind_Find_FDE(t) +   0x39 = 0xcebc0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:1014
            0xcb675            0xcb675  ld.lib.so                       uw_frame_state_for(t) +   0x95 = 0xcb5e0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:419
            0x49720            0x49720  ld.lib.so                       Genode::Allocator_avl_base::Block::find_by_address(unsigned long, unsigned long, bool)(T) +    0x0 = 0x49720
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/allocator_avl.cc:51
            0xd1d4f            0xd1d4f  libc.lib.so                     strtoimax(T) +   0x2f = 0xd1d20
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/strtoimax.c:49
            0x49720            0x49720  ld.lib.so                       Genode::Allocator_avl_base::Block::find_by_address(unsigned long, unsigned long, bool)(T) +    0x0 = 0x49720
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/allocator_avl.cc:51
            0x499a6            0x499a6  libc.lib.so                     pthread_cond_init_int(t) +   0x26 = 0x49980
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:221
            0xcd5f0            0xcd5f0  ld.lib.so                       read_encoded_value_with_base(t) +   0xc0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:248
            0xcd530            0xcd530  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0xcd5f0            0xcd5f0  ld.lib.so                       read_encoded_value_with_base(t) +   0xc0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:248
            0xcb643            0xcb643  ld.lib.so                       uw_frame_state_for(t) +   0x63 = 0xcb5e0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1243
            0xcadb0            0xcadb0  ld.lib.so                       execute_cfa_program(t) +   0xc0 = 0xcacf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:953
            0x4a63e            0x4a63e  ld.lib.so                       Genode::Allocator_avl_base::size_at(void const*) const(T) +   0x3e = 0x4a600
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/allocator_avl.cc:364
            0xcb5e0            0xcb5e0  ld.lib.so                       uw_frame_state_for(t) +    0x0 = 0xcb5e0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1229
            0xcae68            0xcae68  ld.lib.so                       execute_cfa_program(t) +  0x178 = 0xcacf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:956
            0xcb5fb            0xcb5fb  ld.lib.so                       uw_frame_state_for(t) +   0x1b = 0xcb5e0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1234
            0x49b6b            0x49b6b  libc.lib.so                     pthread_key_create_int(t) +   0x4b = 0x49b20
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:225
            0xcab30            0xcab30  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcab30
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0xd2efc            0xd2efc  ld.lib.so                       __gxx_exception_cleanup(_Unwind_Reason_Code, _Unwind_Exception*)(t) +   0x1b = 0xd2ee1
            0xcec35            0xcec35  ld.lib.so                       _Unwind_Find_FDE(t) +   0x75 = 0xcebc0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:1053
            0xcc863            0xcc863  ld.lib.so                       _Unwind_RaiseException_Phase2(t) +   0x43 = 0xcc820
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind.inc:64
            0xcad76            0xcad76  ld.lib.so                       execute_cfa_program(t) +   0x86 = 0xcacf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:974
            0x4b2d0            0x4b2d0  ld.lib.so                       Genode::Avl_node_base::insert(Genode::Avl_node_base*, Genode::Avl_node_base::Policy&)(T) +   0x50 = 0x4b280
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/avl_tree.cc:105
            0xcd530            0xcd530  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0xd228b            0xd228b  ld.lib.so                       parse_lsda_header(_Unwind_Context*, unsigned char const*, lsda_header_info*)(t) +   0x1b = 0xd2270
            0x499a6            0x499a6  libc.lib.so                     pthread_cond_init_int(t) +   0x26 = 0x49980
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:221
            0x499a6            0x499a6  libc.lib.so                     pthread_cond_init_int(t) +   0x26 = 0x49980
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:221
            0x49930            0x49930  libc.lib.so                     pthread_cond_destroy_int(t) +   0x10 = 0x49920
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:220
            0xd1f82            0xd1f82  ld.lib.so                       read_encoded_value_with_base(unsigned char, unsigned long, unsigned char const*, unsigned long*)(t) +   0xa3 = 0xd1edf
            0xd1f82            0xd1f82  ld.lib.so                       read_encoded_value_with_base(unsigned char, unsigned long, unsigned char const*, unsigned long*)(t) +   0xa3 = 0xd1edf
            0xcd5f0            0xcd5f0  ld.lib.so                       read_encoded_value_with_base(t) +   0xc0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:248
            0x7bb91            0x7bb91  libc.lib.so                     Libc::init_mem_alloc(Genode::Env&)(T) +  0x2e1 = 0x7b8b0
                                                                                                      in file /home/sk/src/genode/repos/libports/src/lib/libc/libc_mem_alloc.h:72
            0x499a6            0x499a6  libc.lib.so                     pthread_cond_init_int(t) +   0x26 = 0x49980
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:221
            0x4b77b            0x4b77b  libc.lib.so                     pthread_join_int(t) +   0x2b = 0x4b750
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:273
            0x4bc10            0x4bc10  libc.lib.so                     _spinlock_debug_stub(T) +    0x0 = 0x4bc10
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_spinlock_stub.c:78
            0xce6a4            0xce6a4  libc.lib.so                     r_sort_b(t) +   0x14 = 0xce690
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/radixsort.c:241
            0xa5c80            0xa5c80  ld.lib.so                       Genode::Slab::Block::_slab_entry(int)(T) +   0x20 = 0xa5c60
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/misc_math.h:54
            0x4ac00            0x4ac00  ld.lib.so                       Genode::Avl_node_base::_adopt(Genode::Avl_node_base*, bool, Genode::Avl_node_base::Policy&)(T) +    0x0 = 0x4ac00
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/avl_tree.cc:36
            0x4ac00            0x4ac00  ld.lib.so                       Genode::Avl_node_base::_adopt(Genode::Avl_node_base*, bool, Genode::Avl_node_base::Policy&)(T) +    0x0 = 0x4ac00
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/avl_tree.cc:36
            0xa5c80            0xa5c80  ld.lib.so                       Genode::Slab::Block::_slab_entry(int)(T) +   0x20 = 0xa5c60
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/misc_math.h:54
            0xccdf0            0xccdf0  ld.lib.so                       _Unwind_RaiseException(t) +    0x0 = 0xccdf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind.inc:83
            0xd1bf5            0xd1bf5  ld.lib.so                       base_of_encoded_value(unsigned char, _Unwind_Context*)(t) +    0x0 = 0xd1bf5
            0x49b6b            0x49b6b  libc.lib.so                     pthread_key_create_int(t) +   0x4b = 0x49b20
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:225
            0xcd530            0xcd530  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0xd1f95            0xd1f95  ld.lib.so                       read_encoded_value_with_base(unsigned char, unsigned long, unsigned char const*, unsigned long*)(t) +   0xb6 = 0xd1edf
            0xd0100            0xd0100  ld.lib.so                       memset(W) +    0x0 = 0xd0100
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/cxx/misc.cc:165
            0xcc436            0xcc436  libc.lib.so                     ldiv(T) +   0x26 = 0xcc410
            0x4a336            0x4a336  libc.lib.so                     pthread_rwlock_wrlock_int(t) +   0x46 = 0x4a2f0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:243
            0x4ba3e            0x4ba3e  libc.lib.so                     pthread_setcanceltype_int(t) +   0x4e = 0x4b9f0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:276
            0xd1360            0xd1360  ld.lib.so                       __cxa_free_exception(T) +    0x0 = 0xd1360
            0x80480            0x80480  ld.lib.so                       Genode::Avl_tree<Genode::Allocator_avl_base::Block>::Policy::recompute(Genode::Avl_node_base*)(t) +    0x0 = 0x80480
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/avl_tree.h:203
            0x80480            0x80480  ld.lib.so                       Genode::Avl_tree<Genode::Allocator_avl_base::Block>::Policy::recompute(Genode::Avl_node_base*)(t) +    0x0 = 0x80480
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/avl_tree.h:203
            0xd17c3            0xd17c3  ld.lib.so                       __cxa_begin_catch(T) +   0x48 = 0xd177b
            0xcae68            0xcae68  ld.lib.so                       execute_cfa_program(t) +  0x178 = 0xcacf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:956
            0xd0946            0xd0946  ld.lib.so                       __cxxabiv1::__class_type_info::__do_catch(std::type_info const*, void**, unsigned int) const(t) +   0x2a = 0xd091c
            0x85b00            0x85b00  ld.lib.so                       Genode::Cancelable_lock::unlock()(T) +    0x0 = 0x85b00
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/lock.cc:157
            0xcc9f2            0xcc9f2  ld.lib.so                       uw_install_context_1(t) +   0x42 = 0xcc9b0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1637 (discriminator 2)
            0xd1619            0xd1619  ld.lib.so                       __cxxabiv1::__get_exception_header_from_ue(_Unwind_Exception*)(t) +    0x0 = 0xd1619
            0xd1619            0xd1619  ld.lib.so                       __cxxabiv1::__get_exception_header_from_ue(_Unwind_Exception*)(t) +    0x0 = 0xd1619
            0xcacf0            0xcacf0  ld.lib.so                       execute_cfa_program(t) +    0x0 = 0xcacf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:953
            0xa5c80            0xa5c80  ld.lib.so                       Genode::Slab::Block::_slab_entry(int)(T) +   0x20 = 0xa5c60
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/misc_math.h:54
            0xa9bc5            0xa9bc5  libc.lib.so                     __res_nupdate(T) +   0x15 = 0xa9bb0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/resolv/res_update.c:97
            0xa5c88            0xa5c88  ld.lib.so                       Genode::Slab::Block::_slab_entry(int)(T) +   0x28 = 0xa5c60
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/misc_math.h:55
            0x83676            0x83676  ld.lib.so                       Genode::Heap::_unsynchronized_alloc(unsigned long, void**)(t) +   0xa6 = 0x835d0
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/heap.cc:177
            0xcae68            0xcae68  ld.lib.so                       execute_cfa_program(t) +  0x178 = 0xcacf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:956
            0x7bba1            0x7bba1  libc.lib.so                     Libc::init_mem_alloc(Genode::Env&)(T) +  0x2f1 = 0x7b8b0
                                                                                                      in file /home/sk/src/genode/repos/base/include/base/allocator_avl.h:239
            0xd1f95            0xd1f95  ld.lib.so                       read_encoded_value_with_base(unsigned char, unsigned long, unsigned char const*, unsigned long*)(t) +   0xb6 = 0xd1edf
            0xd1f95            0xd1f95  ld.lib.so                       read_encoded_value_with_base(unsigned char, unsigned long, unsigned char const*, unsigned long*)(t) +   0xb6 = 0xd1edf
            0xa5c80            0xa5c80  ld.lib.so                       Genode::Slab::Block::_slab_entry(int)(T) +   0x20 = 0xa5c60
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/misc_math.h:54
            0xbff90            0xbff90  ld.lib.so                       Genode::stack_area_virtual_base()(t) +    0x0 = 0xbff90
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/stack_area_addr.cc:18
            0xbff90            0xbff90  ld.lib.so                       Genode::stack_area_virtual_base()(t) +    0x0 = 0xbff90
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/stack_area_addr.cc:18
            0xce6a4            0xce6a4  libc.lib.so                     r_sort_b(t) +   0x14 = 0xce690
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/radixsort.c:241
            0xcc65d            0xcc65d  ld.lib.so                       uw_init_context_1(t) +   0x1d = 0xcc640
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1557
            0xa5cd0            0xa5cd0  ld.lib.so                       Genode::Slab::Block::_slab_entry_idx(Genode::Slab::Entry*)(t) +    0x0 = 0xa5cd0
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/slab.cc:161
            0xd2690            0xd2690  ld.lib.so                       __gxx_personality_v0(T) +   0x76 = 0xd261a
            0xce691            0xce691  libc.lib.so                     r_sort_b(t) +    0x1 = 0xce690
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/radixsort.c:241
            0xcd0a3            0xcd0a3  ld.lib.so                       _cxx__Unwind_Resume(t) +   0x73 = 0xcd030
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind.inc:236
            0x80480            0x80480  ld.lib.so                       Genode::Avl_tree<Genode::Allocator_avl_base::Block>::Policy::recompute(Genode::Avl_node_base*)(t) +    0x0 = 0x80480
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/avl_tree.h:203
            0xcad60            0xcad60  ld.lib.so                       execute_cfa_program(t) +   0x70 = 0xcacf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:956
            0xcad60            0xcad60  ld.lib.so                       execute_cfa_program(t) +   0x70 = 0xcacf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:956
            0xcc436            0xcc436  libc.lib.so                     ldiv(T) +   0x26 = 0xcc410
            0xcadb0            0xcadb0  ld.lib.so                       execute_cfa_program(t) +   0xc0 = 0xcacf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:953
            0x80480            0x80480  ld.lib.so                       Genode::Avl_tree<Genode::Allocator_avl_base::Block>::Policy::recompute(Genode::Avl_node_base*)(t) +    0x0 = 0x80480
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/avl_tree.h:203
            0xa5da4            0xa5da4  ld.lib.so                       Genode::Slab::Block::alloc()(T) +   0x74 = 0xa5d30
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/slab.cc:119
            0xd1f82            0xd1f82  ld.lib.so                       read_encoded_value_with_base(unsigned char, unsigned long, unsigned char const*, unsigned long*)(t) +   0xa3 = 0xd1edf
            0xcb832            0xcb832  ld.lib.so                       uw_frame_state_for(t) +  0x252 = 0xcb5e0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:282
            0xcd530            0xcd530  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0xd2b48            0xd2b48  libc.lib.so                     strtoull(T) +   0x78 = 0xd2ad0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/strtoull.c:63
            0xd1f95            0xd1f95  ld.lib.so                       read_encoded_value_with_base(unsigned char, unsigned long, unsigned char const*, unsigned long*)(t) +   0xb6 = 0xd1edf
            0xd2fd4            0xd2fd4  ld.lib.so                       __cxa_throw(T) +   0x65 = 0xd2f6f
            0xd2041            0xd2041  libc.lib.so                     strtol(T) +   0x61 = 0xd1fe0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/strtol.c:65
            0xd1f95            0xd1f95  ld.lib.so                       read_encoded_value_with_base(unsigned char, unsigned long, unsigned char const*, unsigned long*)(t) +   0xb6 = 0xd1edf
            0xce8a0            0xce8a0  libc.lib.so                     r_sort_b(t) +  0x210 = 0xce690
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/radixsort.c:251
            0xcceea            0xcceea  ld.lib.so                       _Unwind_RaiseException(t) +   0xfa = 0xccdf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind.inc:132
            0xa5c80            0xa5c80  ld.lib.so                       Genode::Slab::Block::_slab_entry(int)(T) +   0x20 = 0xa5c60
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/misc_math.h:54
            0x498f0            0x498f0  ld.lib.so                       Genode::Allocator_avl_base::Block::recompute()(T) +    0x0 = 0x498f0
                                                                                                      in file /home/sk/src/genode/repos/base/include/util/avl_tree.h:159
            0x49ddd            0x49ddd  ld.lib.so                       Genode::Allocator_avl_base::free(void*)(T) +   0x4d = 0x49d90
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/allocator_avl.cc:344
            0xcc436            0xcc436  libc.lib.so                     ldiv(T) +   0x26 = 0xcc410
            0xcc436            0xcc436  libc.lib.so                     ldiv(T) +   0x26 = 0xcc410
            0x4aee8            0x4aee8  libc.lib.so                     pthread_attr_setstackaddr_exp(t) +   0x68 = 0x4ae80
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:260
            0xcb5fb            0xcb5fb  ld.lib.so                       uw_frame_state_for(t) +   0x1b = 0xcb5e0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1234
            0xcb5fb            0xcb5fb  ld.lib.so                       uw_frame_state_for(t) +   0x1b = 0xcb5e0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1234
            0x49ac0            0x49ac0  libc.lib.so                     pthread_getspecific_int(t) +    0x0 = 0x49ac0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:224
            0x49f40            0x49f40  ld.lib.so                       Genode::Allocator_avl_base::_cut_from_block(Genode::Allocator_avl_base::Block*, unsigned long, unsigned long, Genode::Allocator_avl_base::Block*, Genode::Allocator_avl_base::Block*)(T) +    0x0 = 0x49f40
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/allocator_avl.cc:148
            0x49f40            0x49f40  ld.lib.so                       Genode::Allocator_avl_base::_cut_from_block(Genode::Allocator_avl_base::Block*, unsigned long, unsigned long, Genode::Allocator_avl_base::Block*, Genode::Allocator_avl_base::Block*)(T) +    0x0 = 0x49f40
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/allocator_avl.cc:148
            0xcd530            0xcd530  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0xd362c            0xd362c  ld.lib.so                       std::type_info::__is_pointer_p() const(t) +    0x0 = 0xd362c
            0xd362c            0xd362c  ld.lib.so                       std::type_info::__is_pointer_p() const(t) +    0x0 = 0xd362c
            0xcd530            0xcd530  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0xac550            0xac550  ld.lib.so                       Genode::Thread::myself()(T) +    0x0 = 0xac550
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/thread_myself.cc:23
            0xce6a4            0xce6a4  libc.lib.so                     r_sort_b(t) +   0x14 = 0xce690
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/radixsort.c:241
            0xce6a4            0xce6a4  libc.lib.so                     r_sort_b(t) +   0x14 = 0xce690
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/radixsort.c:241
            0xd2fd4            0xd2fd4  ld.lib.so                       __cxa_throw(T) +   0x65 = 0xd2f6f
            0xcd530            0xcd530  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0xcd530            0xcd530  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0x4b280            0x4b280  ld.lib.so                       Genode::Avl_node_base::insert(Genode::Avl_node_base*, Genode::Avl_node_base::Policy&)(T) +    0x0 = 0x4b280
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/avl_tree.cc:95
            0xd1f82            0xd1f82  ld.lib.so                       read_encoded_value_with_base(unsigned char, unsigned long, unsigned char const*, unsigned long*)(t) +   0xa3 = 0xd1edf
            0xd1f82            0xd1f82  ld.lib.so                       read_encoded_value_with_base(unsigned char, unsigned long, unsigned char const*, unsigned long*)(t) +   0xa3 = 0xd1edf
            0xce6a4            0xce6a4  libc.lib.so                     r_sort_b(t) +   0x14 = 0xce690
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/radixsort.c:241
            0x4b2d0            0x4b2d0  ld.lib.so                       Genode::Avl_node_base::insert(Genode::Avl_node_base*, Genode::Avl_node_base::Policy&)(T) +   0x50 = 0x4b280
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/avl_tree.cc:105
            0x4b2d0            0x4b2d0  ld.lib.so                       Genode::Avl_node_base::insert(Genode::Avl_node_base*, Genode::Avl_node_base::Policy&)(T) +   0x50 = 0x4b280
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/avl_tree.cc:105
            0xd1f82            0xd1f82  ld.lib.so                       read_encoded_value_with_base(unsigned char, unsigned long, unsigned char const*, unsigned long*)(t) +   0xa3 = 0xd1edf
            0xcc660            0xcc660  ld.lib.so                       uw_init_context_1(t) +   0x20 = 0xcc640
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1562
            0xd27cc            0xd27cc  ld.lib.so                       __gxx_personality_v0(T) +  0x1b2 = 0xd261a
            0xcacf0            0xcacf0  ld.lib.so                       execute_cfa_program(t) +    0x0 = 0xcacf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:953
            0xcacf0            0xcacf0  ld.lib.so                       execute_cfa_program(t) +    0x0 = 0xcacf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:953
            0x4998c            0x4998c  libc.lib.so                     pthread_cond_init_int(t) +    0xc = 0x49980
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:221
            0x4998c            0x4998c  libc.lib.so                     pthread_cond_init_int(t) +    0xc = 0x49980
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:221
            0xcc331            0xcc331  libc.lib.so                     l64a_r(T) +    0x1 = 0xcc330
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/l64a.c:38
            0xbff90            0xbff90  ld.lib.so                       Genode::stack_area_virtual_base()(t) +    0x0 = 0xbff90
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/stack_area_addr.cc:18
            0xcc2b4            0xcc2b4  libc.lib.so                     imaxabs(T) +   0x14 = 0xcc2a0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/imaxabs.c:36
            0xcc2b4            0xcc2b4  libc.lib.so                     imaxabs(T) +   0x14 = 0xcc2a0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/imaxabs.c:36
            0xd1f95            0xd1f95  ld.lib.so                       read_encoded_value_with_base(unsigned char, unsigned long, unsigned char const*, unsigned long*)(t) +   0xb6 = 0xd1edf
            0xcd530            0xcd530  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0xcb133            0xcb133  libc.lib.so                     getopt_internal(t) +   0xa3 = 0xcb090
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/getopt_long.c:397
            0xce650            0xce650  ld.lib.so                       search_object(t) +  0x4b0 = 0xce1a0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2-fde.c:914
            0x49c57            0x49c57  libc.lib.so                     pthread_mutex_destroy_int(t) +    0x7 = 0x49c50
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:228
            0x49c57            0x49c57  libc.lib.so                     pthread_mutex_destroy_int(t) +    0x7 = 0x49c50
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:228
            0xd1f95            0xd1f95  ld.lib.so                       read_encoded_value_with_base(unsigned char, unsigned long, unsigned char const*, unsigned long*)(t) +   0xb6 = 0xd1edf
            0xcd530            0xcd530  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0xcb832            0xcb832  ld.lib.so                       uw_frame_state_for(t) +  0x252 = 0xcb5e0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:282
            0xd2041            0xd2041  libc.lib.so                     strtol(T) +   0x61 = 0xd1fe0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/strtol.c:65
            0xcb1e1            0xcb1e1  libc.lib.so                     getopt_internal(t) +  0x151 = 0xcb090
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/getopt_long.c:424
            0xcb1e1            0xcb1e1  libc.lib.so                     getopt_internal(t) +  0x151 = 0xcb090
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/getopt_long.c:424
            0x49b14            0x49b14  libc.lib.so                     pthread_getspecific_int(t) +   0x54 = 0x49ac0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:224
            0xd266a            0xd266a  ld.lib.so                       __gxx_personality_v0(T) +   0x50 = 0xd261a
            0xcd5f0            0xcd5f0  ld.lib.so                       read_encoded_value_with_base(t) +   0xc0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:248
            0xd226e            0xd226e  ld.lib.so                       read_encoded_value(_Unwind_Context*, unsigned char, unsigned char const*, unsigned long*)(t) +   0x41 = 0xd222d
            0xcb643            0xcb643  ld.lib.so                       uw_frame_state_for(t) +   0x63 = 0xcb5e0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1243
            0xcb643            0xcb643  ld.lib.so                       uw_frame_state_for(t) +   0x63 = 0xcb5e0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1243
            0xcb643            0xcb643  ld.lib.so                       uw_frame_state_for(t) +   0x63 = 0xcb5e0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:1243
            0xcd530            0xcd530  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0xcd530            0xcd530  ld.lib.so                       read_encoded_value_with_base(t) +    0x0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:198
            0xcb6bc            0xcb6bc  ld.lib.so                       uw_frame_state_for(t) +   0xdc = 0xcb5e0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:438
            0xd2c33            0xd2c33  libc.lib.so                     strtoull(T) +  0x163 = 0xd2ad0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/strtoull.c:93
            0xd2c33            0xd2c33  libc.lib.so                     strtoull(T) +  0x163 = 0xd2ad0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/stdlib/strtoull.c:93
            0xd1cfb            0xd1cfb  ld.lib.so                       read_uleb128(unsigned char const*, unsigned long*)(t) +    0x0 = 0xd1cfb
            0x4b2d0            0x4b2d0  ld.lib.so                       Genode::Avl_node_base::insert(Genode::Avl_node_base*, Genode::Avl_node_base::Policy&)(T) +   0x50 = 0x4b280
                                                                                                      in file /home/sk/src/genode/repos/base/src/lib/base/avl_tree.cc:105
            0xcd5f0            0xcd5f0  ld.lib.so                       read_encoded_value_with_base(t) +   0xc0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:248
            0xcd5f0            0xcd5f0  ld.lib.so                       read_encoded_value_with_base(t) +   0xc0 = 0xcd530
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-pe.h:248
            0xcadb0            0xcadb0  ld.lib.so                       execute_cfa_program(t) +   0xc0 = 0xcacf0
                                                                                                      in file /home/krishna/tmp/genode.git/contrib/gcc-e062a6fcfe6dcbfcba8bbb58f833669766487b56/src/noux-pkg/gcc/libgcc/unwind-dw2.c:953
            0x4af0c            0x4af0c  libc.lib.so                     pthread_attr_setstackaddr_int(t) +   0x1c = 0x4aef0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:260
            0x4af0c            0x4af0c  libc.lib.so                     pthread_attr_setstackaddr_int(t) +   0x1c = 0x4aef0
                                                                                                      in file /home/sk/src/genode/contrib/libc-be4908ff7037258395bb0dfbc5d4c5ffef22f859/src/lib/libc/lib/libc/gen/_pthread_stubs.c:260
            0xcae68            0xcae68  ld.lib.so                       execute_cfa_program(t) +  0x178 = 0xcacf0

[skalk]

Ok, further analysis showed that in the special hw signal handling an exception gets thrown that is not used on other kernels at repos/base/src/lib/base/signal_common.cc:195. That exception never gets caught in the error-case of the server component. Normally it should get caught at repos/base/src/lib/base/signal_common.cc:153. The throwing seems to end up in an endless loop. Maybe another exception gets thrown during allocation of the actual exception?

[skalk]

Yes malloc in the cxx library gets called again and again.

[skalk]

My previous observation was half-cocked. It is indeed the case that the faulty component again and again throws an exception, but not because some weird cxx library path. Instead the Libc Entrypoint code calls over and over again the Signal_receiver::pending_signal function, but in non-blocking state. Therefore Signal_receiver::wait_for_signal never gets called, and there is no execution context in base-hw that can receive a signal from the kernel in that case.

[skalk]

Ok, the final outcome is the twofold:

• On hw kernel, if there is a libc component actively polling for io signals without blocking somewhere in the meantime, io signals do not arrive, because there is no second thread putting signals in the corresponding registry like on other kernel. A fix is to ask the kernel for the signals without blocking.
• the VFS lxip plugin does not return the correct return value when accept is called on a socket. It returns READ_OK when it actually should indicate to block. This leads to an active loop as long there is no new connection available.

[skalk]

@chelmuth there is a fixup commit for the signal changes in hw that solves the send/received signal differences on my merge_to_staging branch commit 48e311d1deb2118facd510663b6475a310d07890