Pixracer: Sudden baro pressure and temperature jumps

[mswingtra]

Yesterday in the flight we experienced the following weird behaviour:

When flying in position controlled mode, the plane suddenly started descending between 1-2 metres. After having a look at the logs we saw that the baro had a jump in its measurement of around 10 metres which caused the controller to descend (as it is filtered by the ekf it did not try to drop 10 metres instantly). Changing pixracer and flashing older code version (one prior to the major changes in sensors structure) had the same problem. The pattern is always the same. The baro temperature drops around 0.5°C then rises 1°C and goes back to normal. In the meantime, the baro altitude rises around 10 metres and then drops to normal again. The whole thing takes around 1.5 seconds. Though time wise it can also take longer.

Example:

Logs from two different pixracers: http://logs.uaventure.com/view/eaUNJD4XABW8pxNqDmvWRZ http://logs.uaventure.com/view/NAJGvkrZpUuuy4jHipZgkW

---

Afterwards, we started going through all the previous flights to see whether this has happened before. We have spotted the same pattern on at least 6 different pixracer boards, flown on 4 different aircraft configurations (incl. wingwing) throughout last 3 months. It seem to happen at random points in time. I wonder if anyone else observed this issue.

Majority of the flights were we found this was with baro shielded the way you guys recommended - black foam tightly screwed:

Also, when baro measurements change due to lack of shielding, it looks much different:

We are now logging a long session on the bench to see how often it occurs and whether it depends on environment (though 4 different airframes prove this one wrong), so we'll update when we find out something new.

[bkueng]

Could it be related to https://github.com/PX4/Firmware/issues/5150 ?

[LorenzMeier]

@mswingtra Very likely a candidate issue. Unfortunately the original reporter didn't propose a fix and test - could you have a pass on that as soon as possible?

His last comment describes what to change: You need to implement the last operation from the datasheet correctly. Make sure to send the PR before you flight-test it so I can review the math change.

[LorenzMeier]

I'm not sure this fixes it, but that's what the other issue discussed:

diff --git a/src/drivers/ms5611/ms5611.cpp b/src/drivers/ms5611/ms5611.cpp
index 300c0aa..4997a54 100644
--- a/src/drivers/ms5611/ms5611.cpp
+++ b/src/drivers/ms5611/ms5611.cpp
@@ -693,7 +693,7 @@ MS5611::collect()
        if (_measure_phase == 0) {

                /* temperature offset (in ADC units) */
-               int32_t dT = (int32_t)raw - ((int32_t)_prom.c5_reference_temp << 8);
+               int64_t dT = (int32_t)raw - ((int32_t)_prom.c5_reference_temp << 8);

                /* absolute temperature in centidegrees - note intermediate value is outside 32-bit range */
                _TEMP = 2000 + (int32_t)(((int64_t)dT * _prom.c6_temp_coeff_temp) >> 23);

[CarlOlsson]

I did two flights with the wingwing With current master c76c8fda044e91fb966504131239d83862831a0b

With dT declared as int64_t

I will upload the logs when I get back to wifi

[CarlOlsson]

Logs master fw baro fix

[RomanBapst]

@CarlOlsson @bkueng Out of interest just wondering if anybody has looked at these logs yet?

[LorenzMeier]

@CarlOlsson the log you shared does not include the same temperature range as the original one. Could you please clarify what aspects you were trying to illustrate with these logs? Can you repro a bench test with a hot air gun and the original temperature values exhibiting the issue?

[CarlOlsson]

Hi, the logs from the wingwing was simply just to illustrate what changes in flight with or without the change of declaration. However we found a way to reproduce it to 100% yesterday. Michael will update here as soon as the postprocessing is done

[mswingtra]

@LorenzMeier We have observed that issue yesterday, so we have applied the fix you proposed couple of posts above. However, this has not helped and I am not really surprised because the fix is related to temperature compensation which, according to datasheet, should happen only for temperatures below 20°C.

Anyway, since it was very repetitive, we narrowed down what's triggering it. It has been happening every time we have sent change in parameters over telemetry. We suspect it has to do something with voltage levels across pixracer, so we will investigate it further with logic analyzer later today. Also, we have got RC spikes twice yesterday, and this has also seemed like being triggered by the same thing, just much less often - possibly RC receiver is not that susceptible as baro.

[julianoes]

@LorenzMeier where is this baro fix? It looks to me that this potential fix would only matter in the case where temperature is below 20 deg C.

[mswingtra]

We have done some checks, and long story short, with try and error method, we have found out that the issue has been fixed on PX4 master with commit 021f0840ae7f724bfdfd317dda3d57362017b835. Before that, issue is clearly present, even when no peripherals are connected (except USB and nutshell). Unfortunately for us, our last subtree update was 9 commits older, so we didn't have that fix.

The thing here is that we do order our pixracers directly for AUAV, and we do not have any cheap chinese copies. So, could any of you guys check and confirm that this happens on your pixracers too? Just change any parameter and observe altitude in QGC - for the mentioned commit, and commit before it.

EDIT: And if you could explain quickly how exactly this commit fixes the issue - that'd be great!

[pkocmoud]

@mswingtra I would want to make sure that your issue is not related to https://github.com/PX4/Firmware/issues/5188 as you are interacting with both the baro and FRAM.

[mswingtra]

@pkocmoud It is definitely related! We still observe RC spikes but it does not happen every time we send a param value to the pixracer. This is then quite similar to https://github.com/PX4/Firmware/issues/5170. We're using Taranis transmitter, and FrSky x4r receiver connected on S-BUS, so possibly different setup results in a different result.

Oddly though, Lorenz's commit fixes issue with baro.

[mswingtra]

I am closing here, as the issue with baro jumps is solved.

The RC spikes are still present, but it seems to be related only to our custom changes, as we could not reproduce it with current master.

[nicolaerosia]

I'm also experiencing ms5611 sudden jumps in both temperature and pressure, I'm using Navio2 on RPI v3, master

    if (baro_report.temperature < 20 || baro_report.temperature > 60 || baro_report.altitude > 200) {
        DF_LOG_INFO("%" PRIu64 " ms5611 temp, alt (%f, %f)", baro_report.timestamp, (double)baro_report.temperature, (double)baro_report.altitude);
        perf_end(_baro_sample_perf);
        return 0;
    }

14143394 14142166 ms5611 temp, alt (-250.660004, 16859.552734)
17943426 17942199 ms5611 temp, alt (-250.660004, 30724.326172)
19563741 19562514 ms5611 temp, alt (-250.660004, 17357.546875)
22367662 22366435 ms5611 temp, alt (-63.630001, 2960.404785)
76963587 76962331 ms5611 temp, alt (-250.660004, nan)
113147632 113146404 ms5611 temp, alt (-52.650002, 2355.209229)
118967486 118966257 ms5611 temp, alt (-31.510000, 1430.450195)
132947364 132946138 ms5611 temp, alt (-155.770004, 23740.421875)
148147395 148146167 ms5611 temp, alt (-148.619995, 16656.683594)
149143793 149142566 ms5611 temp, alt (-250.660004, 3718.624023)
150166487 150165259 ms5611 temp, alt (-56.009998, 2531.519043)
161567807 161566579 ms5611 temp, alt (-44.139999, 1948.868042)
162563276 162562049 ms5611 temp, alt (-250.660004, 18654.621094)
166165514 166164288 ms5611 temp, alt (-229.119995, nan)
177166439 177165211 ms5611 temp, alt (-50.090000, 2228.850830)
210343368 210342143 ms5611 temp, alt (-250.660004, nan)

[nicolaerosia]

Taking a look at the raw values, it seems that if the temperature raw value is 0, the pressure value will also be broken, even more, if the pressure value read is zero, the next temperature reading will be broken (non-zero).