Error in ambient pressure compensation for SCD4x sensor

[SimoneBnc]

The problem

After updating to latest esphome 2023.2.2 from latest version of 2022, my SCD41 CO2 sensor started reporting erratic values jumping between zero and several thousand ppm and back to zero. By looking at the logs it seems that the set_ambient_pressure_compensation function sometime it send the correct pressure (from a BME280 sensor) of around 1018hPa then for several times it sends 1 hPa. If I disable the function to use the dynamic compensation and using just the fixed altitude_compensation it works correctly. Below I attached the yaml snippet that work, by removing the comments from the BME280 Pressure - OnValue it will start behaving strangely.

Which version of ESPHome has the issue?

2023.2.2

What type of installation are you using?

Home Assistant Add-on

Which version of Home Assistant has the issue?

2023.2.5

What platform are you using?

ESP32

Board

esp32dev

Component causing the issue

I2C Sensor SCD4x

Example YAML snippet

i2c:
  sda: 21
  scl: 22
  scan: True
  id: bus_a

sensor:
  - platform: scd4x
    co2:
      name: "CO2"
    temperature:
      name: "Temperatura Sala"
    humidity:
      name: "Umidità Sala"
    id: scd41
    i2c_id: bus_a
    altitude_compensation: 140m
    measurement_mode: low_power_periodic
    ambient_pressure_compensation_source: pressione
    update_interval: 60s

  - platform: bme280
    temperature:
      name: "BME280 Temperature"
      oversampling: 16x
    pressure:
      name: "BME280 Pressure"
      id: pressione
#      on_value:
#        then:
#            - lambda: "id(scd41)->set_ambient_pressure_compensation(x / 1000.0);"
    humidity:
      name: "BME280 Humidity"
    address: 0x76
    iir_filter: 16x
    update_interval: 60s

Anything in the logs that might be useful for us?

[10:38:19][D][scd4x:275]: setting ambient pressure compensation to 1010 hPa
[10:38:19][D][sensor:127]: 'CO2': Sending state 0.00000 ppm with 0 decimals of accuracy
[10:38:19][D][sensor:127]: 'Temperatura Sala': Sending state 22.46216 °C with 2 decimals of accuracy
[10:38:19][D][sensor:127]: 'Umidità Sala': Sending state 40.51208 % with 2 decimals of accuracy
[10:38:26][D][sensor:127]: 'BME280 Temperature': Sending state 23.67000 °C with 1 decimals of accuracy
[10:38:26][D][sensor:127]: 'BME280 Pressure': Sending state 1010.08954 hPa with 1 decimals of accuracy
[10:38:26][D][scd4x:275]: setting ambient pressure compensation to 1 hPa
[10:38:26][D][sensor:127]: 'BME280 Humidity': Sending state 39.50391 % with 1 decimals of accuracy
[10:39:19][D][scd4x:275]: setting ambient pressure compensation to 1010 hPa
[10:39:19][D][sensor:127]: 'CO2': Sending state 0.00000 ppm with 0 decimals of accuracy
[10:39:19][D][sensor:127]: 'Temperatura Sala': Sending state 20.90805 °C with 2 decimals of accuracy
[10:39:19][D][sensor:127]: 'Umidità Sala': Sending state 43.84308 % with 2 decimals of accuracy
[10:39:26][D][sensor:127]: 'BME280 Temperature': Sending state 23.67000 °C with 1 decimals of accuracy
[10:39:26][D][sensor:127]: 'BME280 Pressure': Sending state 1010.09033 hPa with 1 decimals of accuracy
[10:39:26][D][scd4x:275]: setting ambient pressure compensation to 1 hPa
[10:39:26][D][sensor:127]: 'BME280 Humidity': Sending state 39.64746 % with 1 decimals of accuracy
[10:40:19][D][scd4x:275]: setting ambient pressure compensation to 1010 hPa
[10:40:19][D][sensor:127]: 'CO2': Sending state 0.00000 ppm with 0 decimals of accuracy
[10:40:19][D][sensor:127]: 'Temperatura Sala': Sending state 20.30724 °C with 2 decimals of accuracy
[10:40:19][D][sensor:127]: 'Umidità Sala': Sending state 45.28656 % with 2 decimals of accuracy
[10:40:26][D][sensor:127]: 'BME280 Temperature': Sending state 23.67000 °C with 1 decimals of accuracy
[10:40:26][D][sensor:127]: 'BME280 Pressure': Sending state 1010.08698 hPa with 1 decimals of accuracy
[10:40:26][D][scd4x:275]: setting ambient pressure compensation to 1 hPa

Additional information

No response

[CarlosGS]

Hi! This must be related to https://github.com/esphome/esphome/pull/4357 Basically the unit for the pressure sensor was changed to hPa.

Unfortunately the change failed to reflect as breaking in the release notes :( And also the documentation improvement didn't get merged automatically: https://github.com/esphome/esphome-docs/pull/2645

Let me know if you can fix it with the new examples. Sorry for the trouble!

[SimoneBnc]

@CarlosGS after removing the division by 1000 in the pressure compensation it works. Thanks. Hopefully the documentation will be updated too.

[Papadoma]

Removing the division by 1000 doesn't work for me. The sensor suddenly gives me values in excess of 14000, I would be almost dead at that concentration. Does it need any time to self calibrate? This only happened after the said update, it was working perfectly before that.

[CarlosGS]

Hmm, can you share the relevant parts of your yaml? I want to understand how that could happen

[SimoneBnc]

@Papadoma this is my current code:

  - platform: bme280
    temperature:
      name: "BME280 Temperature"
      oversampling: 16x
    pressure:
      name: "BME280 Pressure"
      id: pressione
      on_value:
        then:
            - lambda: "id(scd41)->set_ambient_pressure_compensation(x);"

[CarlosGS]

The change in units was made so code can be simplified, you can use directly:

    sensor:
      - platform: bme280
        pressure:
          name: "Ambient Pressure"
          id: bme_pressure
      - platform: scd4x
        id: my_scd41
        measurement_mode: low_power_periodic
        ambient_pressure_compensation_source: bme_pressure
        temperature_offset: 0
        co2:
          name: "CO2 level"

While the lambda method is still available, this is much cleaner and now interoperable with the rest of pressure sensors.

@Papadoma Also note you don't need to specify altitude compensation if using this method (I think the sensor takes preference, but it could be causing conflict). And you are right, the SCD4X has calibration memory, so it could potentially take a bit to stabilize if it has been working with the incorrect pressure for a while after the update. You can speed this up by placing the sensor 5 minutes in clean air and calibrating with ~419ppm. Include this service in the ESPHome code and call it from Home Assistant->Development tools->Services:

    api:
      services:
        - service: calibrate_co2_value
          variables:
            co2_ppm: int
          then:
          - scd4x.perform_forced_calibration:
              value: !lambda 'return co2_ppm;'
              id: my_scd41

Again, I'm sorry for the breaking change. Let me know when you get it working again :)

[Papadoma]

Hmm, can you share the relevant parts of your yaml? I want to understand how that could happen

Sure

  - platform: scd4x
    temperature_offset: 6.8
    id: scd40
    co2:
      name: "SCD40 CO2"
    temperature:
      name: "SCD40 Temperature"
      id: sdc40_temp
    humidity:
      name: "SCD40 Humidity"
      id: sdc40_hum          
    measurement_mode: periodic   
    ambient_pressure_compensation_source: bme280_pressure       

  - platform: bme680_bsec
    temperature:
      name: "BME680 Temperature"
      #internal: True
      id: bme680_temp
    humidity:
      name: "BME680 Humidity"  
      #internal: True
      id: bme680_hum        
    pressure:
      name: "BME680 Pressure"
      id: bme280_pressure
#      on_value:
#        then:
#        - lambda: "id(scd40)->set_ambient_pressure_compensation(x);"       

I tried both with the "ambient_pressure_compensation_source" setting and the "on_value" of the BME680. Currently the CO2 reading is 14,128 ppm 

[CarlosGS]

Indeed it looks like the SCD4X got miscalibrated due to the different pressure input, thinking there is much less concentration and updating the baseline ~400ppm to a wrong value. It should fix automatically in a few days or you can manually calibrate to clean air.

[Papadoma]

In that case, I will report back in about 5-7 days, when the automatic calibration is about to kick in.

[SimoneBnc]

@Papadoma you can force a factory reset and let it auto calibrate, since according to the datasheet you should let it breathe outdoor air for a couple of weeks for at least an hour for initial calibration to take place

[Papadoma]

Indeed it looks like the SCD4X got miscalibrated due to the different pressure input, thinking there is much less concentration and updating the baseline ~400ppm to a wrong value. It should fix automatically in a few days or you can manually calibrate to clean air.

Waited more than 7 days and the value was still wrong. I've triggered the calibration to a fixed value according to the documentation via a template button and for now it seems that the scale is correct, but I have no way of confirming the actual value.

[SimoneBnc]

@Papadoma according to datasheet you should reset it and let it "breathe" outdoor air for at least one hour a day for 2 weeks if you want auto calibration to work

[CarlosGS]

Thanks for getting back, I'm sorry it didn't calibrate in the end :( I've looked into it, and it turns out that in fact if there's no automatic_self_calibration: then it defaults to OFF. Can you check the logs to confirm it prints Automatic self calibration: OFF?

Also you can try factory reset as Simone proposes! Since self-calibration was probably OFF, the original factory calibration "should" make it closer to your original values :thinking:

[Papadoma]

@Papadoma according to datasheet you should reset it and let it "breathe" outdoor air for at least one hour a day for 2 weeks if you want auto calibration to work

Must admit, I didn't check the datasheet. However, leaving it outside for an hour a day for 2 weeks seems unlikely in my situation unfortunately.

Thanks for getting back, I'm sorry it didn't calibrate in the end :( I've looked into it, and it turns out that in fact if there's no automatic_self_calibration: then it defaults to OFF. Can you check the logs to confirm it prints Automatic self calibration: OFF?

Also you can try factory reset as Simone proposes! Since self-calibration was probably OFF, the original factory calibration "should" make it closer to your original values 🤔

The automatic self calibration is reported as ON.

[SimoneBnc]

@Papadoma it is enough to open window of the room. I understand that may be cold outside, but you can still perform a reset later on when weather allows

[CarlosGS]

Oops! I tested it too quickly and came to the wrong conclusion. Then maybe the automatic calibration algorithm has a condition that prevents it from running if the CO2 reading is too high, and since it always read >10.000ppm it would never trigger. Such a condition would make sense to prevent bad calibrations when fresh air isn't available, but would be a problem in this situation.

Now that you have manually calibrated the sensor, it should stabilize itself and it's a matter of observing the overall CO2 readings in the long term. I compared these SCD4Xs with other sensors from different manufacturers, and at different altitudes, and the readings are similar & consistent. As long as you see a clear long-term baseline around 400ppm, your readings should be pretty accurate :+1:

Sorry again for the breaking change, and thanks to both for your help! :)

[SimoneBnc]

@CarlosGS I don't think the readings of more than 10k ppm where an issue, since the onboard microcontroller should only auto calibrate for the first 2 weeks and keep reporting a ppm value, that should be reasonable, the problem is when you add the ambient pressure compensation that with the wrong unit of measurements generates all those wrong numbers. Breaking change on its own it wasn't that bad, the problem was the documentation that did not get updated accordingly and generated confusion (it seems it wasn't your fault)