search

PX4 / PX4-Autopilot

PX4 Autopilot Software
https://px4.io
BSD 3-Clause "New" or "Revised" License
8.21k stars 13.38k forks source link

Exception pause debug in lockstep scheduler in OSX (Visual Studio Code) #12661

Open pedrogasg opened 5 years ago

pedrogasg commented 5 years ago

Describe the bug Running the simulator in debug SITL, the launch start but pauses several times with 3 different errors time_wait, result and `timeout. The simulation is still running but you need to press continue(F5) until the process runs without pausing. This doesn't happen in Ubuntu Bionic (18.04)

To Reproduce Steps to reproduce the behavior:

  1. Install the toolchain for OSX (depending your libraries you may need to install gazebo 9)
  2. Install Visual Studio Code
  3. Clone Firmware
  4. Open Firmware folder with Visual Studio Code
  5. Install recommended extensions for Visual Studio Code
  6. Launch Debug SITL (gazebo iris)
  7. The program pause in pthread_cond_wait

Expected behavior The program shouldn't pause in the wait of a lockstep.

Log Files and Screenshots Debug console log :

Loaded '/usr/lib/system/libsystem_m.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_malloc.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_network.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_networkextension.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_notify.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_sandbox.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_secinit.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_kernel.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_platform.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_pthread.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_symptoms.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libsystem_trace.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libunwind.dylib'. Symbols loaded.
Loaded '/usr/lib/system/libxpc.dylib'. Symbols loaded.
Loaded '/usr/lib/closure/libclosured.dylib'. Symbols loaded.
Loaded '/usr/lib/libobjc.A.dylib'. Symbols loaded.
Loaded '/usr/lib/libc++abi.dylib'. Symbols loaded.
Loaded '/Users/dev/projects/Firmware/build/px4_sitl_default/bin/px4'. Symbols loaded.
=thread-selected,id="2"
Execute debugger commands using "-exec <command>", for example "-exec info registers" will list registers in use (when GDB is the debugger)
The program '/Users/dev/projects/Firmware/build/px4_sitl_default/bin/px4' has exited with code 0 (0x00000000).

backtrace

  thread #1: tid = 0x691a2, 0x00007fff53b5924a libsystem_kernel.dylib`__wait4_nocancel + 10, name = 'px4', queue = 'com.apple.main-thread'
    frame #0: 0x00007fff53b5924a libsystem_kernel.dylib`__wait4_nocancel + 10
    frame #1: 0x00007fff53ad39de libsystem_c.dylib`system + 452
    frame #2: 0x00000001000036c6 px4`main [inlined] run_startup_script(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) + 1490 at main.cpp:520
    frame #3: 0x00000001000030f4 px4`main(argc=0, argv=0x00007ffeefbff8d9) + 5988 at main.cpp:276
    frame #4: 0x00007fff53a08015 libdyld.dylib`start + 1
    frame #5: 0x00007fff53a08015 libdyld.dylib`start + 1
  thread #3: tid = 0x691cf, 0x00007fff53b5806e libsystem_kernel.dylib`__close_nocancel + 10
    frame #0: 0x00007fff53b5806e libsystem_kernel.dylib`__close_nocancel + 10
    frame #1: 0x00007fff53a930e9 libsystem_c.dylib`fclose + 92
    frame #2: 0x0000000100188a3f px4`px4_daemon::Server::_server_main(this=<unavailable>) + 719 at server.cpp:215
    frame #3: 0x0000000100188766 px4`px4_daemon::Server::_server_main_trampoline(self=<unavailable>) + 6 at server.cpp:120
    frame #4: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #5: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #6: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #4: tid = 0x691d0, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x0000000100189949 px4`LockstepScheduler::cond_timedwait(this=0x00000001005001f0, cond=0x00000001007002d0, lock=0x0000000100700290, time_us=10000000) + 185 at lockstep_scheduler.cpp:88
    frame #3: 0x000000010012774b px4`DriverFramework::HRTWorkQueue::process(this=<unavailable>) + 283 at DriverFramework.cpp:486
    frame #4: 0x000000010012761f px4`DriverFramework::HRTWorkQueue::process_trampoline(arg=<unavailable>) + 415 at DriverFramework.cpp:361
    frame #5: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #6: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #7: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #5: tid = 0x691d1, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'hpwork'
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x0000000100189949 px4`LockstepScheduler::cond_timedwait(this=0x00000001005001f0, cond=0x000070000635ce38, lock=0x000070000635ce68, time_us=962000) + 185 at lockstep_scheduler.cpp:88
    frame #3: 0x0000000100189abf px4`LockstepScheduler::usleep_until(this=<unavailable>, time_us=<unavailable>) + 191 at lockstep_scheduler.cpp:124
    frame #4: 0x000000010012d41a px4`work_hpthread(argc=<unavailable>, argv=<unavailable>) + 26 at work_thread.c:270
    frame #5: 0x000000010012a952 px4`entry_adapter(ptr=0x0000000100700000) + 66 at px4_posix_tasks.cpp:105
    frame #6: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #7: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #8: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #6: tid = 0x691d2, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'lpwork'
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x0000000100189949 px4`LockstepScheduler::cond_timedwait(this=0x00000001005001f0, cond=0x00007000063dfe38, lock=0x00007000063dfe68, time_us=962000) + 185 at lockstep_scheduler.cpp:88
    frame #3: 0x0000000100189abf px4`LockstepScheduler::usleep_until(this=<unavailable>, time_us=<unavailable>) + 191 at lockstep_scheduler.cpp:124
    frame #4: 0x000000010012d43d px4`work_lpthread(argc=<unavailable>, argv=<unavailable>) + 29 at work_thread.c:296
    frame #5: 0x000000010012a952 px4`entry_adapter(ptr=0x0000000100700030) + 66 at px4_posix_tasks.cpp:105
    frame #6: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #7: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #8: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  * frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x0000000100189949 px4`LockstepScheduler::cond_timedwait(this=0x00000001005001f0, cond=0x0000700006462e08, lock=0x0000700006462e38, time_us=1884000) + 185 at lockstep_scheduler.cpp:88
    frame #3: 0x0000000100189abf px4`LockstepScheduler::usleep_until(this=<unavailable>, time_us=<unavailable>) + 191 at lockstep_scheduler.cpp:124
    frame #4: 0x000000010012d01f px4`work_hrtthread [inlined] hrt_work_process + 20 at hrt_thread.c:222
    frame #5: 0x000000010012d00b px4`work_hrtthread(argc=<unavailable>, argv=<unavailable>) + 27 at hrt_thread.c:266
    frame #6: 0x000000010012a952 px4`entry_adapter(ptr=0x0000000100700370) + 66 at px4_posix_tasks.cpp:105
    frame #7: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #8: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #9: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #7: tid = 0x691d4, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'wq:manager'
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x000000010012b491 px4`::px4_sem_wait(s=0x0000000100600208) + 65 at px4_sem.cpp:89
    frame #3: 0x0000000100186010 px4`px4::WorkQueueManagerRun() [inlined] BlockingQueue<px4::wq_config_t const*, 1ul>::pop(this=0x0000000100600190) + 9 at BlockingQueue.hpp:69
    frame #4: 0x0000000100186007 px4`px4::WorkQueueManagerRun() + 295 at WorkQueueManager.cpp:181
    frame #5: 0x000000010012a952 px4`entry_adapter(ptr=0x00000001007004a0) + 66 at px4_posix_tasks.cpp:105
    frame #6: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #7: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #8: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #8: tid = 0x691f7, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'dataman'
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x000000010012b491 px4`::px4_sem_wait(s=0x000000010020a210) + 65 at px4_sem.cpp:89
    frame #3: 0x000000010003ff7a px4`task_main(argc=<unavailable>, argv=<unavailable>) + 650 at dataman.cpp:1368
    frame #4: 0x000000010012a952 px4`entry_adapter(ptr=0x00000001020002b0) + 66 at px4_posix_tasks.cpp:105
    frame #5: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #6: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #7: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #9: tid = 0x691fe, 0x00007fff53b5a0a2 libsystem_kernel.dylib`poll + 10, name = 'sim_rcv'
    frame #0: 0x00007fff53b5a0a2 libsystem_kernel.dylib`poll + 10
    frame #1: 0x00000001000d8fe2 px4`Simulator::poll_for_MAVLink_messages(this=<unavailable>) + 1186 at simulator_mavlink.cpp:775
    frame #2: 0x00000001000d6319 px4`Simulator::start(argc=4, argv=0x0000000100402c10) + 169 at simulator.cpp:114
    frame #3: 0x000000010012a952 px4`entry_adapter(ptr=0x0000000100402bf0) + 66 at px4_posix_tasks.cpp:105
    frame #4: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #5: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #6: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #10: tid = 0x691ff, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'sim_send'
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x0000000100189949 px4`LockstepScheduler::cond_timedwait(this=0x00000001005001f0, cond=0x00007000066f1d08, lock=0x00007000066f1cc8, time_us=1000000) + 185 at lockstep_scheduler.cpp:88
    frame #3: 0x000000010012b588 px4`::px4_sem_timedwait(s=0x00007000066f1cc8, abstime=0x00007000066f1c90) + 136 at px4_sem.cpp:137
    frame #4: 0x0000000100184cd8 px4`::px4_poll(fds=<unavailable>, nfds=<unavailable>, timeout=100) + 600 at cdev_platform.cpp:384
    frame #5: 0x00000001000d8962 px4`Simulator::send(this=0x0000000102800600) + 274 at simulator_mavlink.cpp:602
    frame #6: 0x00000001000d8850 px4`Simulator::sending_trampoline((null)=<unavailable>) + 16 at simulator_mavlink.cpp:578
    frame #7: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #8: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #9: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #11: tid = 0x6920d, 0x00007fff53b58d8a libsystem_kernel.dylib`__semwait_signal + 10
    frame #0: 0x00007fff53b58d8a libsystem_kernel.dylib`__semwait_signal + 10
    frame #1: 0x00007fff53ad3724 libsystem_c.dylib`nanosleep + 199
    frame #2: 0x00007fff53ad3618 libsystem_c.dylib`usleep + 53
    frame #3: 0x00000001001899ea px4`LockstepScheduler::TimedWait::~TimedWait() [inlined] LockstepScheduler::TimedWait::~TimedWait(this=0x00000001007008b0) + 38 at lockstep_scheduler.h:43
    frame #4: 0x00000001001899c4 px4`LockstepScheduler::TimedWait::~TimedWait(this=0x00000001007008b0) + 4 at lockstep_scheduler.h:23
    frame #5: 0x00007fff53a0ad59 libdyld.dylib`tlv_finalize + 49
    frame #6: 0x00007fff53d22163 libsystem_pthread.dylib`_pthread_tsd_cleanup + 463
    frame #7: 0x00007fff53d21ee9 libsystem_pthread.dylib`_pthread_exit + 79
    frame #8: 0x00007fff53d2066c libsystem_pthread.dylib`_pthread_body + 351
    frame #9: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #10: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13
  thread #12: tid = 0x6920e, 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'wq:hp_default'
    frame #0: 0x00007fff53b58a1e libsystem_kernel.dylib`__psynch_cvwait + 10
    frame #1: 0x00007fff53d21589 libsystem_pthread.dylib`_pthread_cond_wait + 732
    frame #2: 0x000000010012b491 px4`::px4_sem_wait(s=0x0000700006774e50) + 65 at px4_sem.cpp:89
    frame #3: 0x0000000100185b78 px4`px4::WorkQueue::Run(this=0x0000700006774dc0) + 56 at WorkQueue.cpp:108
    frame #4: 0x000000010018638c px4`px4::WorkQueueRunner(context=<unavailable>) + 92 at WorkQueueManager.cpp:166
    frame #5: 0x00007fff53d20661 libsystem_pthread.dylib`_pthread_body + 340
    frame #6: 0x00007fff53d2050d libsystem_pthread.dylib`_pthread_start + 377
    frame #7: 0x00007fff53d1fbf9 libsystem_pthread.dylib`thread_start + 13

Capture d’écran 2019-08-08 à 14 42 39

Capture d’écran 2019-08-08 à 14 44 59

Drone (please complete the following information):

Additional context Tested in different versions of OSX

julianoes commented 5 years ago

I tried to reproduce this with make px4_sitl gazebo___lldb and could not see it. I'll have to set-up vscode on macOS and try it as well.

julianoes commented 4 years ago

@pedrogasg have you found out more about this? I can reproduce an exception pause on macOS now but it doesn't seem to be the same backtrace from what I can tell.

Also see: https://discuss.px4.io/t/vscode-lldb-debugging-paused-on-exception/13512/7?u=julianoes

pedrogasg commented 4 years ago

Hello, I actually start to working in a linux machine after this, my last try was to set the miDebuggerPath to lldb-mi, but I have some issue so I give up for the moment, if some is working to solve this maybe can follow this track...

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.