Vehicle becomes unresponsive after issuing micrortps_client stop command

[adk4davsys]

Describe the bug Today during the position mode hover, issuing a command micrortps_client stop while the client was running before arming the vehicle caused it to become unresponsive to any inputs. We tried to switch the flight mode to manual, back to position, and to RTL using the transmitter, as well as setting it to land from QGC. The vehicle just hovered until it flipped over and crashed. Luckily, there were no injuries or significant damages occurred since it was only hovering at about 2m.

To Reproduce To reproduce this, I tried to do the same thing in a HITL simulation. Steps to reproduce the behavior in HITL:

1. Start a HITL with gazebo Tools/sitl_gazebo/worlds/hitl_iris.world.
2. On a separate terminal, start the agent on companion computer via SSH: micrortps_agent -d /dev/ttyS0.

   Note: this crash can be reproduced without running the agent

3. On the MAVLink Console in QGC, start the client: micrortps_client start -d /dev/ttyS2.
4. Using a transmitter, switch to position mode and takeoff to a certain altitude.
5. Back on MAVLink Console in QGC, stop the client: micrortps_client stop
6. QGC loses the connection and the vehicle crashes with the terminal running the HITL gazebo returning TX queue overflow repeatedly.

Expected behavior The vehicle becoming unresponsive to any kind of input from either the transmitter or QGC.

Log Files and Screenshots Always provide a link to the flight log file: PX4 Flight Review.

Drone (please complete the following information):

• Generic quadrotor airframe.

Additional context The physical system was running on v1.12.0-beta5 and HITL were tested using both the beta5 and v1.12.0 release.

[TSC21]

Thanks for reporting. Needs investigation. But the question would also be: why would you stop the client on mid-flight?

[adk4davsys]

Thanks for reporting. Needs investigation. But the question would also be: why would you stop the client on mid-flight?

Thank you for your reply. I did not stop the client mid-flight, it was stopped before arming the vehicle. Also, in HITL, stopping the client while doing nothing loses the connection to vehicle making it unresponsive as well. Please let me know if there is anything else you need to know.

[TSC21]

Also can you please provide a log from that flight? There's also the possibility of an hardfault, for which a log is also generated and is welcomed.

[TSC21]

Thanks for reporting. Needs investigation. But the question would also be: why would you stop the client on mid-flight?

Thank you for your reply. I did not stop the client mid-flight, it was stopped before arming the vehicle. Also, in HITL, stopping the client while doing nothing loses the connection to vehicle making it unresponsive as well. Please let me know if there is anything else you need to know.

I don't get it. So the vehicle became unresponsive but you still were able to arm and takeoff? And if it was unresponsive, why did it takeoff?

[adk4davsys]

Also can you please provide a log from that flight? There's also the possibility of an hardfault, for which a log is also generated

Thanks for reporting. Needs investigation. But the question would also be: why would you stop the client on mid-flight?

Thank you for your reply. I did not stop the client mid-flight, it was stopped before arming the vehicle. Also, in HITL, stopping the client while doing nothing loses the connection to vehicle making it unresponsive as well. Please let me know if there is anything else you need to know.

I don't get it. So the vehicle became unresponsive but you still were able to arm and takeoff? And if it was unresponsive, why did it takeoff?

Sorry for being unclear. I issued the stop command, armed the vehicle with position mode, then took off to an altitude. Then the vehicle became unresponsive. Also to note that on the ground with props off, issuing a stop command sometimes will reboot the vehicle.

And the log is included in my original report. Let me know if the link is broken.

[TSC21]

Also can you please provide a log from that flight? There's also the possibility of an hardfault, for which a log is also generated

Thanks for reporting. Needs investigation. But the question would also be: why would you stop the client on mid-flight?

Thank you for your reply. I did not stop the client mid-flight, it was stopped before arming the vehicle. Also, in HITL, stopping the client while doing nothing loses the connection to vehicle making it unresponsive as well. Please let me know if there is anything else you need to know.

I don't get it. So the vehicle became unresponsive but you still were able to arm and takeoff? And if it was unresponsive, why did it takeoff?

Sorry for being unclear, I issued the stop command, armed the vehicle with position mode, then took off to an altitude. Then the vehicle became unresponsive. Also to note that on the ground with props off, issuing a stop command sometimes will reboot the vehicle.

And the log is included in my original report. Let me know if the link is broken.

From the log you provided, there's a lot of invalid setpoints and EKF resets, but nothing pointing to the micrortps_client being the responsible for the crash, specially considering you were able to takeoff with the vehicle. So I don't think that there's anything pointing to the client being the responsible.

Anyway, what I will investigate is the possible reboots (resulting from harfaults), because I saw that in the past, and thought I had it solved. But again, I don't see any relation to what lead to your crash.

[adk4davsys]

What I don't understand is that I can reproduce the exact same crashing behavior in HITL only when I issue the stop command (Unresponsiveness and falling out of the sky soon after commanding micrortps_client stop in MAVLink Console. Could you please try this on your setup just to make sure this issue is only local to our system?

[TSC21]

What I don't understand is that I can reproduce the exact same crashing behavior in HITL only when I issue the stop command (Unresponsiveness and falling out of the sky soon after commanding micrortps_client stop in MAVLink Console. Could you please try this on your setup just to make sure this issue is only local to our system?

I will try soon enough. But if the vehicle is already flying and then it hardfaults, it's quite natural that then it crashes. But that is different from what I see on your log, because if you say that you stopped the client before arming, then it means that you were flying for almost a minute before it actually crashed. It's kinda difficult to relate the problems to be honest.

[adk4davsys]

I will continue to dig into this to pin point the issue of our crash. Thank you for your time.