d-ronin / dRonin

The dRonin flight controller software.
http://dronin.org
Other
289 stars 167 forks source link

F3/F4 targets have higher CPU usage, likely related to telemetry changes. #2200

Open glowtape opened 6 years ago

glowtape commented 6 years ago

Several users have been mentioning higher CPU usage on the F3 targets lately. Most recent reporter Maquiabelo, where it seems to be causing a critical CPU alarm, and also claims to be unable to arm (presumably due to critical CPU). Two quads using a micro F3, running the Omnibus F3 target.

The UAVOs are attached to this issue, but I haven't looked into it yet. spc95.zip

I'm seeing something similar. When I install an older build on my SPRF3e, it idles at 67%, and with next, it's at 80%. With erased settings both times (LQG, FlipOver and Motorgain don't seem to be responsible for the brunt of it).

I'm currently busy identifying a bunch of CPU time sinks, which may or may not happen only with the GCS attached. It seems like two commits are responsible for the bump on my SPRF3e, each worth around 5%:

https://github.com/d-ronin/dRonin/commit/b0b70c5b79feb6fa09a271bcbe21006cb4d64425 telemetry: throttled obj can only be in queue once https://github.com/d-ronin/dRonin/commit/5388fe3e5a86f43d75de4d264dc63c2df8e2308c uavobjectmanager: improve timer math

The issue with latter one was identified over on IRC, altho it brings the additional CPU from 4-5% down to 1.5-2%, not nearly zero, so there's still a minimal effect on that one.

Former seems unclear.

Maquiabelo has an OSD on his Omnibus. I figure its presence seems to send things over the edge together.

glowtape commented 6 years ago

Eh, I spot this only now, https://github.com/d-ronin/dRonin/commit/5388fe3e5a86f43d75de4d264dc63c2df8e2308c is a small patch applying to https://github.com/d-ronin/dRonin/commit/b0b70c5b79feb6fa09a271bcbe21006cb4d64425 .

glowtape commented 6 years ago

Yeah I don't get it. Right now on latest next, it idles at 77% with no config at all. When I revert the recent change 263720787679d638bc3257be998a5ae16f877497, which gives back 3%, and the other two changes above, I drop to 70%. I've been looking and prodding at it, and I don't get it.

glowtape commented 6 years ago

OK, what I've noticed is that per TaskInfo (reverted vs next):

Actuator 12% -> 13% Attitude 26% -> 29% Stabilization 28% -> 30% ManualControl "0%" -> 1%

The interesting thing is that the job these tasks do is very definite and on a timer. The amount of UAVO accesses should not vary per tick, neither do the changes look like they'd do more work. If at all, they'd do less, because throttled queues aren't potentially updated as much.

glowtape commented 6 years ago

Ugh, it has an effect on F4, too. I went to do some logging with some lean custom logging profile, the sort that didn't use to flash the CPU warning (at 166hz), but now it does (at 100hz).

So I've installed the LQG test build from the forums, the CPU usage on my Seppuku was this, logging disabled though:

1) Only board powered, no receiver: 50% 2) Only board powered, receiver on, armed in LQG rate: 55% 2) Whole quad powered (for the OSD IRQ to hog CPU), receiver on, armed in LQG rate: 70-71%

With next:

1) 57% 2) 62-63% 3) 78-79%

With the things above reverted:

1) 52% 2) 57-58% 3) 72-73%

I figure some remaining overhead in the third case comes from the vertical thrust stuff and probably moving BMI160 from interrupts to polling, that weren't present in the forum build.

glowtape commented 6 years ago

So next with #2210 brings some slight improvements, the CPU usage got slightly lower.

1) 55% 2) 61% 3) 76-77%

glowtape commented 6 years ago

2210 doesn't seem to have an effect on SPRF3e idle CPU usage, which stays at 76-77% regardless.

glowtape commented 6 years ago

Latest changes in #2210 bring F3 idle down to 71-72%, whereas with the reverts, it's 70-71%. So that definitely did something.

As for the Seppuku: 1) 51% 2) 56-57% 3) 72%

It's practically fixed, but it appears we're optimizing other things to make up for it, since PIOS_Thread_Systime() was always in use. Later today I'm starting with next and am going to knock out things in the commits above on next bit by bit to find what triggers it.

glowtape commented 6 years ago

Nevermind, PIOS_Thread_Period_Elapsed() is new.

glowtape commented 6 years ago

OK, I think I've tracked it down. I've knocked out everything by commenting out, CPU usage stayed high.

The mere commenting of this line dropped the CPU from 77% to 72%, even though that pointer wasn't in use anymore, because of having commented out other things using it beforehand. https://github.com/d-ronin/dRonin/blob/next/flight/UAVObjects/inc/uavobjectmanager.h#L120

glowtape commented 6 years ago

With #2210 and #2211, SPRF3e idle CPU usage went to 68-69%. It used to be 67% a few weeks back, but more stuff went into stab.c meanwhile.

As for Seppuku: 1) 50% 2) 55% 3) 69-71%