Compensation algorithm causes high spikes in measures

[el-han]

The compensation algorithm in this library uses shift operations in order to efficiently achieve a division by powers of 2. However, these two operations are not guaranteed to be equivalent when applied to signed integers.

When using this library on an atmega328p, I often get high spikes in the resulting temperature and humidity readings. These spikes disappear when I use divisions instead of bit shifts.

I will propose a PR which uses the reference implementation of the compensation algorithm from Bosch. This implementation uses division operations. Could you have a look at it and maybe test it on different platforms?

[caternuson]

Can you provide an example that demonstrates the issue? Or provide more information under what conditions the issue occurs.

[el-han]

Difficult to nail the exact conditions where the spikes occur. I have experienced them especially during winter when I open the Window and the indoor conditions (e.g. temperature and humidity) change rapidly. Here is an example of temperature (degrees C) and humidity readings of one day: Although the temperature in this example seems to have spikes up to one specific temperatire, I have also experienced some temperature spikes around 100, above 300 and slightly above 0 degrees C.

[ventilator]

Similar to my experience: Setup: 20 ESPs with 20 Bosch BMEs, each measuring 1 set of points (temperature, humidity, pressure) each minute, then go into deep sleep for one minute. Most of the measurements data is fine, but many spikes many sensors each hour or so. This does not happen so often if the deep sleep of the esp is disabled. I guess additionally some issue with the starup of the sensor from powerloss?

I censored the names of the sensors, but each graph shows 20 different BME280s In this setup all sensors are on the same table (22°C, but every now and then -145°C is reported)

Code snippets: `

include

    #include <SPI.h>
    #include <Adafruit_Sensor.h>
    #include <Adafruit_BME280.h>
    Adafruit_BME280 bme; // I2C

    delay(1);
    if (bme_sensorID < 255) { //bme_sensorID is 96 if sensor is connected, 255 if no sensor was found during init. readTemperature hangs forever if there is no sensor
        bme.takeForcedMeasurement(); // take one measurement, go to sleep after it

        temperature = bme.readTemperature();
        pressure = bme.readPressure();
        humidity = bme.readHumidity();`

[ladyada]

does it happen with other libraries or just this one?

[el-han]

It also happens with modm which also uses shift operations to calculate the compensated values:

https://github.com/modm-io/modm/blob/develop/src/modm/driver/pressure/bme280_data_impl_fp.hpp#L69

[ladyada]

what bout https://github.com/sparkfun/SparkFun_BME280_Arduino_Library?

[el-han]

Let me check. It might take a while since under the current weather conditions the spikes have become quite rare.

[Kiotheka]

Hello, I may have found the problem that causes the spikes and I have a solution.

[Kiotheka]

Datasheet Bosch BST-BME280-DS002.pdf Page 23

4. Data readout

To read out data after a conversion, itis strongly recommended to use a burst read and not address every register individually. This will prevent a possible mix-up of bytes belonging to different measurements and reduce interface traffic. Note that in I2C mode, even when pressure was not measured, reading the unused registers is faster than reading temperature and humidity data. separately.

Data readout is done by starting a burst read from 0xF7 to OXFC (temperature and pressure) or from OXF7 to OXFE (temperature, pressure and humidity). The data are read out in an unsigned 20-bit format both for pressure and for temperature and in an unsigned 16-bit format for humidity. Itis strongly recommended to use the BME280 API, available from Bosch Sensortec, for readout and compensation. For details on memory map and interfaces, please consult chapters 5 and 6 respectively.

After the uncompensated values for pressure, temperature and humidity ‘ut, ‘up’ and ‘uh’ have been read, the actual humidity, pressure and temperature needs to be calculated using the compensation parameters stored in the device. The procedure is elaborated in chapter 4.2.

4.1 Data register Shadowing

In normal mode, the timing of measurements is not necessarily synchronized to the readout by the user. This means that new measurement results may become available while the user is reading the results from the previous measurement. In this case, shadowing is performed in order to guarantee data consistency. Shadowing will only work if all data registers are read in a single burst read. Therefore, the user must use burst reads if he does not synchronize data readout with the measurement cycle. Using several independent read commands may result in inconsistent data.

If a new measurement s finished and the data registers are still being read, the new measurement results are transferred into shadow data registers. The content of shadow registers is transferred into data registers as soon as the user ends the burst read, even if not all data registers were read.

The end of the burst read is marked by the rising edge of CSB pin in SPI case or by the recognition of a stop condition in I2C case. After the end of the burst read, all user data registers are updated at once.

[Kiotheka]

uint8_t Adafruit_BME280::read64(byte reg, int32_t &valuet, int32_t &valuep, int32_t &valueh) {

  if (_cs == -1) {
    _wire->beginTransmission((uint8_t)_i2caddr);
    _wire->write((uint8_t)reg);
    _wire->endTransmission();
    _wire->requestFrom((uint8_t)_i2caddr, (byte)8);

    valuep = _wire->read();
    valuep <<= 8;
    valuep |= _wire->read();
    valuep <<= 8;
    valuep |= _wire->read();

    valuet = _wire->read();
    valuet <<= 8;
    valuet |= _wire->read();
    valuet <<= 8;
    valuet |= _wire->read();

    valueh = _wire->read();
    valueh <<= 8;
    valueh |= _wire->read();

    return 1;
  } else {
    if (_sck == -1)
      _spi->beginTransaction(SPISettings(500000, MSBFIRST, SPI_MODE0));
    digitalWrite(_cs, LOW);
    spixfer(reg | 0x80); // read, bit 7 high

    valuep = spixfer(0);
    valuep <<= 8;
    valuep |= spixfer(0);
    valuep <<= 8;
    valuep |= spixfer(0);

    valuet = spixfer(0);
    valuet <<= 8;
    valuet |= spixfer(0);
    valuet <<= 8;
    valuet |= spixfer(0);

    valueh = spixfer(0);
    valueh <<= 8;
    valueh |= spixfer(0);
    digitalWrite(_cs, HIGH);
    if (_sck == -1) {
      _spi->endTransaction(); // release the SPI bus
      return 1;
    }
  }
  return 0;
}

void Adafruit_BME280::read_TPH(float &temperature, float &pressure, float &humidity)
{

  int32_t adc_T;
  int32_t adc_P;
  int32_t adc_H;
  int32_t var1t, var2t;
  int64_t var1p, var2p, p;

  read64(BME280_REGISTER_PRESSUREDATA, adc_T, adc_P, adc_H);

  // Temperature
  if (adc_T == 0x800000) // value in case temp measurement was disabled
    temperature = NAN;
  adc_T >>= 4;

  var1t = ((((adc_T >> 3) - ((int32_t)_bme280_calib.dig_T1 << 1))) *
           ((int32_t)_bme280_calib.dig_T2)) >>
          11;

  var2t = (((((adc_T >> 4) - ((int32_t)_bme280_calib.dig_T1)) *
             ((adc_T >> 4) - ((int32_t)_bme280_calib.dig_T1))) >>
            12) *
           ((int32_t)_bme280_calib.dig_T3)) >>
          14;

  t_fine = var1t + var2t;

  float T = (t_fine * 5 + 128) >> 8;
  temperature =  T / 100;
  //Pressure
  if (adc_P == 0x800000) // value in case pressure measurement was disabled
    pressure = NAN;
  adc_P >>= 4;

  var1p = ((int64_t)t_fine) - 128000;
  var2p = var1p * var1p * (int64_t)_bme280_calib.dig_P6;
  var2p = var2p + ((var1p * (int64_t)_bme280_calib.dig_P5) << 17);
  var2p = var2p + (((int64_t)_bme280_calib.dig_P4) << 35);
  var1p = ((var1p * var1p * (int64_t)_bme280_calib.dig_P3) >> 8) +
          ((var1p * (int64_t)_bme280_calib.dig_P2) << 12);
  var1p =
    (((((int64_t)1) << 47) + var1p)) * ((int64_t)_bme280_calib.dig_P1) >> 33;

  if (var1p == 0) {
    pressure = NAN; // avoid exception caused by division by zero
  }
  p = 1048576 - adc_P;
  p = (((p << 31) - var2p) * 3125) / var1p;
  var1p = (((int64_t)_bme280_calib.dig_P9) * (p >> 13) * (p >> 13)) >> 25;
  var2p = (((int64_t)_bme280_calib.dig_P8) * p) >> 19;

  p = ((p + var1p + var2p) >> 8) + (((int64_t)_bme280_calib.dig_P7) << 4);
  pressure = (float)p / 256;
  // Humidity
  if (adc_H == 0x8000) // value in case humidity measurement was disabled
    humidity = NAN;

  int32_t v_x1_u32r;

  v_x1_u32r = (t_fine - ((int32_t)76800));

  v_x1_u32r = (((((adc_H << 14) - (((int32_t)_bme280_calib.dig_H4) << 20) -
                  (((int32_t)_bme280_calib.dig_H5) * v_x1_u32r)) +
                 ((int32_t)16384)) >>
                15) *
               (((((((v_x1_u32r * ((int32_t)_bme280_calib.dig_H6)) >> 10) *
                    (((v_x1_u32r * ((int32_t)_bme280_calib.dig_H3)) >> 11) +
                     ((int32_t)32768))) >>
                   10) +
                  ((int32_t)2097152)) *
                 ((int32_t)_bme280_calib.dig_H2) +
                 8192) >>
                14));

  v_x1_u32r = (v_x1_u32r - (((((v_x1_u32r >> 15) * (v_x1_u32r >> 15)) >> 7) *
                             ((int32_t)_bme280_calib.dig_H1)) >>
                            4));

  v_x1_u32r = (v_x1_u32r < 0) ? 0 : v_x1_u32r;
  v_x1_u32r = (v_x1_u32r > 419430400) ? 419430400 : v_x1_u32r;
  float h = (v_x1_u32r >> 12);
  humidity = h / 1024.0;

}

[Kiotheka]

Public
void read_TPH(float &temperature,float &pressure, float &humidity);

protected
uint8_t read64(byte reg, int32_t &valuet, int32_t &valuep, int32_t &valueh);

Using

float temperature;
float pressure;
float humidity;

bme.read_TPH(temperature,pressure,humidity);
Serial.print("T: ");
Serial.print(temperature);
Serial.print("C P: ");
Serial.print(pressure / 100.0F);
Serial.print("hPa RH: "); 
Serial.print(humidity);
Serial.println("%"); 

[Kiotheka]

Hi, This is the solution I use for divisions for signed 32bits, to optimize the code for small microcontrollers.

signed int32 divShiftSigned32(signed int32 valor, int8 shift)
{
  if (valor & 0x80000000)
  {
    valor = (~valor) + 1;
    valor >>= shift;
    return ((~valor) + 1);
  }
  return valor >>= shift;
}

Example of use

BME280_S16_t BME280_compensate_T_int32_A(BME280_U32_t adc_T)
{
BME280_S32_t var1, var2, T;
var1 = divShiftSigned32(((((adc_T>>3) - ((BME280_S32_t)dig_T1 <<1))) * ((BME280_S32_t)dig_T2)),11);// / 2048;
var2 = divShiftSigned32( (divShiftSigned32((((adc_T>>4) - ((BME280_S32_t)dig_T1)) * ((adc_T>>4) - ((BME280_S32_t)dig_T1))),12) * ((BME280_S32_t)dig_T3)),14);// / 16384;
t_fine = var1 + var2;
T = (BME280_S16_t)(divShiftSigned32((t_fine * 5 + 128),8));// /256;
return T;
}

BME280_U32_t BME280_readPressure(int32 adc_P)
{
  signed int32 var1,var1p4, var2;
  int32 p;
  // calculate pressure
  var1 = divShiftSigned32(((signed int32)t_fine),1) - (signed int32)64000;
  var1p4 = divShiftSigned32(var1,2);
  var2 = divShiftSigned32(((var1p4) * (var1p4)) ,11 ) * ((signed int32)dig_P6);

  var2 = var2 + ((var1 * ((signed int32)dig_P5)) * 2);
  var2 = divShiftSigned32(var2,2) + (((signed int32)dig_P4) * 65536);

  var1 = (divShiftSigned32(((signed int32)dig_P3 * divShiftSigned32(((var1p4) * (var1p4)),13 )),3) +
         ((((signed int32)dig_P2) * var1)/2)) / 262144;

  var1 =divShiftSigned32((((32768 + var1)) * ((signed int32)dig_P1)),15);// / 32768;

  if (var1 == 0)
    return 0; // avoid exception caused by division by zero

  p = (((int32)(((signed int32)1048576) - adc_P) - divShiftSigned32(var2 ,12))) * 3125;

  if (p < 0x80000000)
    p = (p * 2) / ((int32)var1);//unsog

  else
    p = (p / (int32)var1) * 2;

  var1 = divShiftSigned32((((int32)dig_P9) * divShiftSigned32((signed int32)(((p/8) * (p/8))),13)),12);// / 4096;
  var2 =  divShiftSigned32((((int32)(p/4)) * ((signed int32)dig_P8)),13);// / 8192;
  p = (int32)((signed int32)p + divShiftSigned32((var1 + var2 + (signed int32)dig_P7),4));
 return p;
}

BME280_U32_t BME280_readHumidity(int32 adc_H)
{
  signed int32 v_x1_u32r;
  v_x1_u32r = (t_fine - ((signed int32)76800));

  v_x1_u32r = ( divShiftSigned32((((adc_H * 16384) - (((signed int32)dig_H4) * 1048576) - (((signed int32)dig_H5) * v_x1_u32r)) +
      ((signed int32)16384)),15) * divShiftSigned32(((divShiftSigned32((divShiftSigned32((v_x1_u32r * ((signed int32)dig_H6)) ,10) * (divShiftSigned32((v_x1_u32r *
      ((signed int32)dig_H3)),11) + ((signed int32)32768))),10) + ((signed int32)2097152)) *
      ((signed int32)dig_H2) + 8192), 14));

  v_x1_u32r = (v_x1_u32r - divShiftSigned32((divShiftSigned32((divShiftSigned32(v_x1_u32r,15) * divShiftSigned32(v_x1_u32r ,15)),7) * ((signed int32)dig_H1)),4));
  v_x1_u32r = (v_x1_u32r < 0 ? 0 : v_x1_u32r);
  v_x1_u32r = (v_x1_u32r > 419430400 ? 419430400 : v_x1_u32r); 
  return  (int32)(v_x1_u32r/4194.304);  
}

[Kiotheka]

adc_P and adc_P are >> 4 before entering the function.

Example adc_P = adc_P>>4; adc_T = adc_T>>4;

[el-han]

@Kiotheka so you think the spikes could be caused by not performing a burst read but reading bytes from different measures? Could be. But doesn't _wire->requestFrom((uint8_t)_i2caddr, (byte)8); request 8 bytes at once and performs a burst read over I²C? Or is it a sequence of 8 single byte reads?

[el-han]

signed int32 divShiftSigned32(signed int32 valor, int8 shift) { if (valor & 0x80000000) { valor = (~valor) + 1; valor >>= shift; return ((~valor) + 1); } return valor >>= shift; }

As far as I understand, the shift operators are implementation specific, so it is generally not defined if they are arithmetic or logic shifts (Please correct me if I'm wrong. The world would be so much simpler!). If you take for example 0x80000000 as input value for your algorithm and shift it by one, it would be recognized as negative, negated (0x3fffffff) and added by one (0x80000000). The next line is a right shift that could be either arithmetic (0xc0000000) or logic (0x40000000).

The first case would return ~(0xc0000000) + 1 which is 0x40000000 (a positive), the second case would result in ~(0x40000000) + 1 which is 0xc0000000 (the correct value).

So same problem here. Since Arduino supports multiple architectures (AVR, several ARMs, ESP, maybe ever Risc-V?) I am not sure if each of them translates the >> operator in the same manner.

[el-han]

@ladyada no spikes so far with the Sparkfun library. Even though in summer spikes are rare I think I should have experienced at least one by now. @ventilator is it possible for you to test this library and report if it makes your spikes disappear too?

[Kiotheka]

@Kiotheka so you think the spikes could be caused by not performing a burst read but reading bytes from different measures? Could be. But doesn't _wire->requestFrom((uint8_t)_i2caddr, (byte)8); request 8 bytes at once and performs a burst read over I²C? Or is it a sequence of 8 single byte reads?

The reading must be 8 bytes at once.

[tom1422]

Getting the same issue, I will also see if the sparfun library helps

[floatAsNeeded]

Hi All! Is there also a way with this library to make the BME680 sleep or something after the reading? Because at the moment i'm using it with an ESP32 and when I go to sleep it consumes 1.5mA ! The esp32 and the other circuit alone without the sensor just use less than 100 microA

[caternuson]

@floatAsNeeded This library is for the BME280, not the BME680. Also, this issue thread is for something else. Please open another issue.

[caternuson]

For anyone seeing this issue, please try the 2.2.0 release of the library and see if the helps.

[FStefanni]

Hi,

I am using the latest library version. Here what I have found in my tests:

1. The sensor is OK, its values are similar to another one (DHT22)
2. The new library seems fine too: no more spikes in 24h :+1:
3. It seems also important to respect the suggested sampling rates (please refer to the datasheet) in order to have good results
   • Suggested rate is 1/60Hz (1m) for weather

So to me this issue seems fixed (but I'll update you the next Monday to see if actually no more spikes have been found -- so we have a long running test).

Regards.

[FStefanni]

Hi,

I confirm that with the latest version, I have no more spikes (almost 6 days till now).

Regards

[caternuson]

OK, thanks for updating. Going to close. Hopefully the changes made for the 2.2.0 release fixed this.