CCS811 calibration problem with 2.x plus

[pabigot]

There seems to be something wrong with the way the Thingy 2.1.0 firmware manages CCS811 calibration.

I've set up several Thingies next to each other in a interior space several meters from a forced air heating vent, connected to a central that receives temperature and humidity every 15 s and gas readings every 10 s and logs the readings to OpenTSDB. The image below covers about 19 hours and demonstrates that the TVOC readings are correlated with temperature with an implausibly high gain.

The data collection started about 16 hours prior to the data displayed in the image with three devices located in three different rooms spanning 8 m maximum distance with no doors blocking air exchange. All devices started at zero TVOC, then rose with slightly different slopes not obviously correlated with the differences in ambient temperature. (Several devices had been used for short periods with the Android app and I suspect the difference was due to previous gas sensor collections.)

The first couple hours in the image show waves in TVOC that rise at times when a forced-hot-air gas furnace was running, and fall when it was off. Use of the furnace explains variation in TVOC, but the magnitude of the reading differences is implausible.

Around 16:00 I moved two devices to location of the device with the lowest magnitude TVOC. This area was about 0.5 Cel warmer than the others which may partly explain the initial ramp-up; around 16:30 the set temperature of the furnace increased, providing an additional significant rise stabilizing around 17:00, after which temperature and magnitude TVOC remained steady for a couple hours.

Around 19:00 I added a fourth device with virtually no prior on-time, which required that the central be restarted. On reconnection all readings dropped to zero, then rose linearly at a rate proportional to their relative difference prior to restart, then at 30 minute jumped significantly. The newly added device, which had no significant prior run-time, read zero for most of this period.

The set temperature for the environment dropped about 5 Cel at 22:00, which is masked by a steady drop after the sudden jump up at 16:30, then rose again around 04:00, with pulses again correlating to furnace activity. During this period the newly-added device (shown in detail in the right column) showed large increases in TVOC gain as temperature.

While all readings are still within the nominal 48-hour burn-in, the trends are not good. The jump at 30 min after connection to a central, as well as the correlation of gain with temperatures, suggest something done by the firmware. The two devices with lowest readings had some burn-in time with firmware 1.1.0; the two with highest readings were delivered with firmware 2.0.0 and immediately updated to 2.1.0.

In short, the as-implemented behavior is horribly wrong. I'll keep this running for a while longer to see if it converges to consistency, but I'm not hopeful. I'm aware of issue #11 which might contribute to fixing this, but I suspect there's something wrong with the on-going maintenance of the BASELINE register as well as absence of specific calibration support. (Both notes on page 11 of the datasheet appear relevant.)

[pabigot]

Further investigation confirms that accurate results for the first 30 minutes after startup are because the persisted BASELINE isn't installed until that interval has passed, and that the persisted values in the two older units is much smaller. And that it doesn't matter, because over time the reading value can grow to the maximum expressed by the sensor and fall again with no clear cause.

At a minimum a way to reset the baseline as a fix for #11 seems justified, but there may be a more serious problem with ongoing tuning that will then have to be addressed. As currently implemented the CCS811 doesn't seem usable, at least not with the devices I have.

[koffes]

Thank you for this highly detailed feedback. We will have to look into this issue in detail in order to provide you with valuable feedback. A few key issues:

• We have run the gas sensors for an extended period of time (>100 hours) and we see decent data coming from the sensor. Please attempt to run the sensors for an even longer period of time and post the results.
• If you update the firmware, the previously stored baseline value will be deleted. This happens since the entire user flash area is deleted upon FW upgrade in Thingy:52.
• If the sensor has been used before, it will warm up for 30 minutes before the old stored baseline is written to it. There should be no additional writes to the gas sensor after the one and only baseline write.
• It seems you are using the the sensor close to a output vent/furnace. Mind that the gas sensor is best applied in e.g. an office environment with fixed/predictable daily fluctuations in CO2 and TVOC.

[pabigot]

I have data from a subsequent 4 day run that shows the same extreme fluctuations. I suspect it's because the baseline calibration is completely wrong. I will try to the flash update process to clear the value and collect another run, probably calibrating in a controlled environment, but won't get to it for a while.

Two of the sensors were above head height on a wall that's about about 2-3 m from a forced hot air furnace output at floor level in rooms ranging from 4-10 m^2. Not particularly close and I wouldn't expect anomalies if the gain weren't so high. The third sensor was desk level 4 m from an output in a 14 m^2 room; it showed the same long-term fluctuations but the short term heating-related ones were not as visible.

In the first image on the right side the temperature and humidity were synchronously collected from one device, with a 5-minute min/avg/max displayed. The Thingy52 is showing about a 0.5 Cel variation in temperature and 3 %RH absolute variation in humidity between consecutive readings. Other co-located sensors (SHT21) show significantly better repeatability (+/- 0.02 Ce, +/- 0.1 %RH). One possibility is that the noise in the temperature and humidity values affects the gas sensor calibration (I believe the gas sensor is given those values, though I haven't reviewed the code to see how often).

[koffes]

Note that the user values in flash will only be deleted if the FW version is different to the one that was stored earlier. So 2.1.0 -> 2.1.0 will retain the values. In order to reset the user flash, do a 2.1.0 -> 2.0.0 ->2.1.0. Do you have access to a 52 development kit or a Segger? In which case this can be done a lot faster using the 10-pin debug port on the Thingy:52.

[pabigot]

Thanks. I have a JLink Plus with the appropriate cable and will use that to just rewrite the 2.1.0 firmware with an intervening --eraseall which should clear the configuration.

[pabigot]

Attached are overview and detail images from a second run with four units, all of which underwent nrfjprog --eraseall followed by a flash of thingy_v2.1.0.HW_1.0.hex downloaded from the product page. All devices had at least 48 hours CCS811 burn-in prior to this experiment, and the calibration value should have been cleared. The image below displays mostly ongoing 10 s CCS811 collection.

All devices displayed implausible readings during the period, though some of the changes may be due to highly magnified real environmental changes. Interestingly the two devices that particularly spiked are from batch 2017.37 purchased in early 20178-12, while the more moderate devices are from batch 2017.20 purchased in early 20178-07. Perhaps a change in build process or parts?

The devices were placed in a box and left alone, except for a short period starting at 2018-01-03T10:50:00 when they were moved to a different room. This resulted in an unexpected rise, and they were returned to their original position within an hour. The collection program was disabled from about 2018-01T11:58 to 2018-01-03T12:15 which reset the BLE connection and sensor readings, but subsequently the connection was continuous on all devices until 2018-01-05T06:35:00 when the devices were moved within the room and something broke the connection to two of them. Detail of the restart suggests if a calibration was stored in the first couple hours its restoration around 12:45 had little impact.

The noise in 15 s temperature/humidity readings is also displayed in the overview.

I'm fairly convinced there's something wrong with the way the sensor is being maintained, but without diving into the code and running experiments with additional instrumentation can't say what it is.

[koffes]

Once again, thank you for this very detailed feedback. We are most curious by your first graph where similar units with identical FW seem to behave differently. I have prioritised this issue in our task system,

[matt7aylor]

Hi, I am also seeing some gas sensor calibration issues with Thingy devices, in particular a similar strong temperature (and humidity) dependence in the reported values. I think the calibration of the CCS811 is likely to be the culprit here but fortunately it seems there should be some relatively easy fixes to explore that may help the issue.

In particular looking through the CCS811 datasheet http://ams.com/eng/content/download/951091/2269479/471718, there is a mechanism in the sensor for temperature and humidity compensation:

"If an external sensor is available this information can be written to CCS811 so that they will be used to compensate gas readings due to temperature and humidity changes. Refer to the ENV_DATA (Environment Data) Register (0x05)."

From a brief analysis of the Thingy 2.1 firmware source it seems that, while there is some code written for this purpose, it is not currently being used e.g. (source/modules/m_environment.c):

/**@brief Sends the sampled humidity and temperature to the gas sensor for calibration.
 *
 * @note Not currently used.
 */
static uint32_t calibrate_gas_sensor(uint16_t humid, float temp)
{

I suspect that adding in the temperature and humidity compensation could make a big difference to this issue. I'm not sure why the written code is not being used, maybe there is a good reason, but it seems like it may just be a matter of grabbing the data being produced by the other sensors.

This might also help with differences you are seeing between units, as Thingys that have been exposed to different temperature/humidity will record very different baselines. For example, if I take a one of my Thingys to a colder environment for a short time, then back to a warmer one (maybe 10C difference) the Thingy CO2 will read 5000+ppm for a ~24 hour period until the automatic baseline correction kicks in (see datasheet) and it might suddenly drop back down to 500ppm. Even smaller 1C variation against a nominal room temperature can produce big (~10%) changes in reported CO2. So adding the temperature and humidity compensation could also help improve general baseline calibration and variation between units.

[pabigot]

Adding temperature compensation is certainly worth investigating, but I'm not convinced it's going to explain or solve everything:

• The data in my comment from 2018-01-06 was collected with all four units together in a small, open-topped box. The environment was the same[*], but inter-device variance for TVOC was high (and correlated with device build date).
• I've subsequently confirmed that the Thingy52 HTS221 is subject to huge inter-reading variation on the ADC values that are used to calculate temperature and humidity. This variance is also visible in that data set. If the CCS811 is sensitive to 1 Cel changes, well, the HTS221 with default 16x oversampling will vary as much as 1 Cel between readings at 1 Hz.

I'm currently running firmware that increases the oversampling rate for HTS221 to 256x, which reduces the noise but still leaves its repeatability far worse than that of a SHT21-based system. HTS221 problems are a different issue, though.

[*] That the environment was the same is not immediately clear from the graphs, which for Temperature and Humidity include other sensors. The four Thingy52's are grouped at the top of the Temperature graph, and in a band around 30% in the Humidity graph. The offsets within each group are essentially constant per-device, so are likely due to them being uncalibrated; the evidence does not support differences between device environment.

[matt7aylor]

Agree.

I think adding the temperature and humidity compensation should hopefully have a big effect on improving some aspects of the calibration though (at least on the temperature dependence issue), certainly worth testing, the gas sensor is almost unusable without it if the environment changes even a small amount. It's interesting that you see such large variation in temperature readings though (at first glance mine seem more like +/-0.1C but I haven't investigated further) this could, as you say, make calibration more challenging.

[hackfin]

Hi there,

My experience with the CS811 is not Thingy-related, but I am also having a hard time getting this sensor to work reliably. For one thing, I see quite some drifting even in pulsed mode in a totally untouched room (H&T constant), as attached, there's some sawtoothy behaviour corresponding to the baseline recalibration by the internal firmware (which is yet obscure to me). See 24h-recordings of ECO2 output, Baseline and Raw value below.

In this range of variation, the artifacts are somewhat ok to me. However when inducing rapid changes of humidity, the firmware seems to go nuts, as somewhat documented here: http://section5.ch/index.php/2018/02/08/ccs811-sensor-review/ I had absolutely no luck with humidity compensation, the baseline and raw values would jump around chaotically between min&max, so the result is unusable.

I've gone through the appnotes several times, but since neither AMS responded to questions so far nor did I get detailed insight on the deployed firmware yet, I am unable to get a proper calibration done, in fact a bare metal SnO2 sensor device is showing more reproduceable results. I'm wondering if I just got hold of a bad sensor (engineering 'alpha' sample) or if the unit got irreversibly tainted by some real bad garlic breath. I currently don't have another 1:1 unit to correlate to.

Don't know if this helps...I have had a first go at disassembling the 8051 firmware out of despair, if someone has gone down that road I'd be interested in sharing details, unless that sensor is really meant to be kicked into the corner...

[matt7aylor]

Over the last week or so I've been testing a few Thingy52s with some very basic Temperature/Humidity compensation added to the firmware: https://github.com/matt7aylor/Nordic-Thingy52-FW/commit/57ed6adfae3faa3054c9ef82b9c953e1fd69f1cc (I've hardcoded sending environment data every 30s when temp/humidity sensing is on and I set my Thingys to measure temp/humidity every 10s and gas every 60s. To avoid getting some weird readings from the devices after firmware upgrade it seems to help to run an nrfjprog --eraseall followed by turning off and de-powering the device for a few seconds before reconnecting and flashing the firmware. Just to make double sure the baseline is properly forgotten in flash and the CCS811 chip itself.)

So far I'd say that the compensation data has helped with the sensitivity to slower variation in temperature and humidity, certainly not a cure for all the above issues though. In most of my sensors adding the compensation seems highlight much more the 'volatility' in the VOC readings. In my, admittedly non-clean room, test environment I typically see eCO2 readings on devices that look like a large fast spike followed by a slower decline, however, it seems that all devices will spike in this way at pretty much the same time (but often with vastly different amplitudes) so what I observe seems likely due to real effects. (Eat an orange near the sensor and it will surely spike massively in 'eCO2' but this is expected behaviour. Some spikes are a mystery though.)

I'd agree that the sensors don't seem to like rapid environment changes, seems to play havoc with the calibration and while the spikes from real effects I see may match up in time between devices the absolute values can be quite different, as can the general background. Both of these may have some relation to the baseline calibration but, as has been pointed out, this seems somewhat obtuse. I think temp/humidity compensation is a step in the right direction though at least for relatively slow varying environments.

[joao-ladeira]

Did anyone figured this out? I was also facing this problem, but i think today i figured out what is going on.

In my case this always happened in the morning after i leave home, at first i thought that maybe was related to the fact that i opened the windows for the air to circulate thus humidity changes more rapidly as well as temperature, and the CCS811 had some kind of weird bug, i tried with 3 sensors and they were all behaving the same.

But today i had one of those aha moment's, it turns out that just before i leave home i put on some deodorant lol this in fact has ALLOT of CO2, so here it is... just wanted to share my two cents.

[valdowb]

Ccs811 sensor is a joke