MAV2 signing-enabled multiple connection instability

[Jaaaky]

Bug report

Issue details

I've noticed some signing issues months ago using MissionPlanner, but didn't have a clue. Today, I could reproduce the bug using mavproxy as the following;

1. Connect to USB using mavproxy on one terminal, run signing setup somepassword. Keep it running
2. On another terminal run mavproxy - I've added signing key somepassword to startup script - connecting to telemetry - in my case using RFD at 57600 BAUD -
3. Notice how slow, intermittent, unstable the connection is on the telemetry connection. The banner is never displayed or displayed too many times with many "no link" & "link 1 down" messages.
4. Go to mavproxy to USB terminal run signing remove.
5. Back to mavproxy to telemetry; close it, reopen, the connection works fine now.

Just make sure mavproxy is connected to USB 1st and keep it connected.

Version Using git bisect I've found that it's an old bug introduced at commit ef46acda711879804b511eb6afec3919e8228809 .. The last good commit is ef3654e3432ec1727d2ab448ad23514facd5698b

I've tested using ArduPlane-master .. The bug doesn't exist for plane3.9 as the buggy change is not cherry-picked there.

Hardware type What autopilot hardware was used? (Pixhawk, Cube, Pixracer, Navio2, etc) Pixhawk1, RFD telemetry

[peterbarker]

@Jaaaky I see something similar without any signing involved. I did find several (non-security-critical) bugs to do with signing, but that's another story.

Could you try the same thing without signing involved, please?

On the telemetry radio, run module load messagerate followed by repeat add 1 messagerate status - and then play with introducing MAVProxy and removing it again. See if the message rates go up and down.

[peterbarker]

My problem is that we get 300us every 20ms (@50Hz) to send messages across all links - so a total of 15,000us/second to send our mavlink messages.

MAVProxy streams at 4Hz, and there are ~25 messages stream in plane - so 100 messages per second.

Some messages take 200us to send. So we can send 70 of those per second - clearly most messages take less than 200us to send or we wouldn't keep up with a single link.

Moving SCHED_LOOP_RATE to 300Hz on Plane fixes the problem I'm seeing - we now get 300us every 3.3ms, so 90,000us every second to send messages.

Raising the 300us (specified in the scheduler table) is bad as it may lead to scheduler overruns when running at higher scheduler loop rates.

This problem will go away once we're using a thread to send messages.

Investigations continue.

[Jaaaky]

Any updates on this? It's really annoying and any telemetry connection with signing is slow and sluggish in-flight. It makes signing almost unusable.

[Jaaaky]

@peterbarker I really can notice that not only sending messages is sluggish but also receiving any mavlink msg too.