Bosch Twinguard exposes only CO2 instead of VOC levels

[burmistrzak]

Hi folks, I've been preparing to finally migrate a handful of Bosch Twinguards to Z2M and saw that the current implementation exposes the VOC sensor readings as CO2 (ppm) instead of VOC (µg/m³).

While @safakaltun did absolutely stellar work, reverse engineering Bosch's ZCL extension, their choice to expose the VOC readings as CO2-equivalent (?) doesn't seem quite right to me.

• First, the mentioned data sheet for the BME680 sensor explicitly states that the sensor cannot directly measure CO2 levels (p. 25), instead having to estimate them.
• Second, it's a bit unclear, to me at least, how @safakaltun came up with that ppm conversion formula. I guess they tried matching the value from the Bosch Smart Home app? 🤔
• And third, exposing only a CO2 (and no VOC) sensor in Z2M, has some unintended consequences for downstream systems like homebridge-z2m. Namely e.g. a missing VOC sensor in HomeKit, something the native integration from Bosch (via their Smart Home Controller) already supports.

Now, what to do about this situation? Well, simply exposing the raw AQI value isn't super useful because, among other things, there are many different, national AQI scales out there. AFAICT, the Twinguard uses the German AQI internally.

@Koenkk What do you think about exposing an additional VOC level, so to not break backwards compatibility while adding to the overall usefulness of the device? 👀

[Koenkk]

Im fine with that!

[burmistrzak]

Im fine with that!

Great!

I'm already trying to get more details from the manufacturer regarding the reported AQI values, how to correctly interpret, and convert them into the appropriate units for VOC.

[burmistrzak]

@Koenkk Alright, got one Twinguard paired. Went surprisingly smooth! However, I wasn't able to /get the current value for alarm and self_test, as you can see below.

Error 2024-03-28 22:20:12Publish 'get' 'alarm' to 'Dachzimmer/Twinguard' failed: 'Error: Unhandled key toZigbee.bosch_twinguard.convertGet alarm'
Error 2024-03-28 22:20:13Publish 'get' 'self_test' to 'Dachzimmer/Twinguard' failed: 'Error: Unhandled key toZigbee.bosch_twinguard.convertGet self_test'

And sure enough, cases for both keys are not implemented. Probably because we don't know yet, how to actually query for them? 🤔

Edit: Hmm, at least self_test is mostly fire & forget, so /get can't work. For alarm, it should be possible to get the current state.

[safakaltun]

Hi, “alarm” is an action. Doesn’t have a state until you trigger it. Once you do that, you will see “siren state” changing accordingly.

[safakaltun]

Current “voc” sensor in z2m has a different unit that ppm. Twinguard uses ppm as voc unit in the bosch app. It also directly reports the voc ppm value. Using co2 sensor was just a workaround to get it implemented into z2m without a lot of hassle. Not 100% correct, but didn’t really matter back then whether it was called co2 or voc or a banana. Happy that you are fixing it :)

[burmistrzak]

Hi, “alarm” is an action. Doesn’t have a state until you trigger it. Once you do that, you will see “siren state” changing accordingly.

Hey, glad to hear from you directly! Yeah, you're correct regarding alarm. We'll have to fix access properties for self_test and alarm then. Both shouldn't allow /get requests (i.e. ea.STATE_SET and ea.SET respectively). That'll also address the errors with the UI mentioned above.

Edit: Might be a silly idea, but couldn't we just do a read on alarm_status to refresh siren_state & Co.? Something like:

        convertGet: async (entity, key, meta) => {
            switch (key) {
            case 'sensitivity':
                await entity.read('manuSpecificBosch', [0x4003], manufacturerOptions);
                break;
            case 'pre_alarm':
                await entity.read('manuSpecificBosch5', [0x4001], manufacturerOptions);
                break;
            case 'heartbeat':
                await meta.device.getEndpoint(12).read('manuSpecificBosch7', [0x5005], manufacturerOptions);
                break;
            case 'alarm': // NEW
            case 'self_test': // NEW
                await meta.device.getEndpoint(12).read('manuSpecificBosch8', [0x5000], manufacturerOptions); // NEW
                break;
            default: // Unknown key
                throw new Error(`Unhandled key toZigbee.bosch_twinguard.convertGet ${key}`);
            }
        },

Edit#2: Well, that worked. Quickly hacked my running Z2M and it's now behaving as described in the docs. 😅

[burmistrzak]

Current “voc” sensor in z2m has a different unit that ppm. Twinguard uses ppm as voc unit in the bosch app. It also directly reports the voc ppm value.

Yup, Bosch is certainly doing some funny business here. Their own data sheet explicitly states, that the BME680 sensor is incapable of directly measuring CO2 levels. They absolutely have to do some calculations on their end, to produce a ppm value from just AQI.

result.co2 = msg.data['airpurity'] * 10.0 + 500.0; On that note, how'd you come up with this specific conversion formula?

[burmistrzak]

@safakaltun And while we have you here, is there a reason for just using manuSpecificBosch3 on endpoint 3 to get all our sensor readings?

From what I can tell, there are dedicated msTemperatureMeasurement, msRelativeHumidity, msPressureMeasurement, etc. clusters available, and all seemingly provide useable data... 🤔

Edit: I was able to add pressure sensor readings with a slightly modified definition (node_modules/zigbee-herdsman-converters/devices/bosch.js) and manually configuring reporting for that cluster.

{
        zigbeeModel: ['Champion'],
        model: '8750001213',
        vendor: 'Bosch',
        description: 'Twinguard',
        fromZigbee: [fromZigbee_1.default.pressure, fzLocal.bosch_twinguard_measurements, fzLocal.bosch_twinguard_sensitivity,
            fzLocal.bosch_twinguard_pre_alarm, fzLocal.bosch_twinguard_alarm_state, fzLocal.bosch_twinguard_smoke_alarm_state,
            fzLocal.bosch_twinguard_heartbeat],
        toZigbee: [tzLocal.bosch_twinguard],
        configure: async (device, coordinatorEndpoint, logger) => {
            const coordinatorEndpointB = coordinatorEndpoint.getDevice().getEndpoint(1);
            await reporting.bind(device.getEndpoint(1), coordinatorEndpointB, [0x0009]);
            await reporting.bind(device.getEndpoint(7), coordinatorEndpointB, [0x0019]);
            await reporting.bind(device.getEndpoint(7), coordinatorEndpointB, [0x0020]);
            await reporting.bind(device.getEndpoint(1), coordinatorEndpointB, [0xe000]);
            await reporting.bind(device.getEndpoint(3), coordinatorEndpointB, [0xe002]);
            await reporting.bind(device.getEndpoint(1), coordinatorEndpointB, [0xe004]);
            await reporting.bind(device.getEndpoint(12), coordinatorEndpointB, [0xe006]);
            await reporting.bind(device.getEndpoint(12), coordinatorEndpointB, [0xe007]);
            await reporting.bind(device.getEndpoint(8), coordinatorEndpointB, ['msPressureMeasurement']);
            await device.getEndpoint(1).read('manuSpecificBosch5', ['unknown_attribute'], manufacturerOptions); // Needed for pairing
            await device.getEndpoint(12).command('manuSpecificBosch7', 'pairingCompleted', manufacturerOptions); // Needed for pairing
            await device.getEndpoint(1).write('manuSpecificBosch', { 0x4003: { value: 0x0002, type: 0x21 } }, manufacturerOptions); // Setting defaults
            await device.getEndpoint(1).write('manuSpecificBosch5', { 0x4001: { value: 0x01, type: 0x18 } }, manufacturerOptions); // Setting defaults
            await device.getEndpoint(12).write('manuSpecificBosch7', { 0x5005: { value: 0x01, type: 0x18 } }, manufacturerOptions); // Setting defaults
            await device.getEndpoint(1).read('manuSpecificBosch', ['sensitivity'], manufacturerOptions);
            await device.getEndpoint(1).read('manuSpecificBosch5', ['pre_alarm'], manufacturerOptions);
            await device.getEndpoint(12).read('manuSpecificBosch7', ['heartbeat'], manufacturerOptions);
        },
        exposes: [
            e.pressure(), e.smoke(), e.temperature(), e.humidity(), e.co2(), e.illuminance_lux(), e.battery(),
            e.enum('alarm', ea.ALL, Object.keys(sirenState)).withDescription('Mode of the alarm (sound effect)'),
            e.text('siren_state', ea.STATE).withDescription('Siren state'),
            e.binary('self_test', ea.ALL, true, false).withDescription('Initiate self-test'),
            e.enum('sensitivity', ea.ALL, Object.keys(smokeSensitivity)).withDescription('Sensitivity of the smoke alarm'),
            e.enum('pre_alarm', ea.ALL, Object.keys(stateOffOn)).withDescription('Enable/disable pre-alarm'),
            e.enum('heartbeat', ea.ALL, Object.keys(stateOffOn)).withDescription('Enable/disable heartbeat'),
        ],
    },

[burmistrzak]

@Koenkk Are you cool with a single omnibus PR with all Twinguard-related commits included? Otherwise, I'll prepare my bugfixes first. 🐛

[burmistrzak]

Not sure about the pressure sensor readings, though... They seem to be way off (e.g. 27388 hPa) , but that's something to worry about later, I guess. 🙄

Anyhow, I think I've made some progress with regards to a VOC conversion formula. On page 9 in the BME680 data sheet, just below the AQI table, there's the following footnote:

According to the guidelines issued by the German Federal Environmental Agency, exceeding 25 mg/m³ of total VOC leads to headaches and further neurotoxic impact on health. The BSEC software auto-calibrates the low and high concentrations applied during testing to IAQ of 50 and 200, respectively.

Looking at the AQI table and comparing the stated health impact, 25 mg/m³ seems to correspond to an AQI of 351. Assuming linear scaling, a reported AQI of 1 should therefore be roughly equal to 71 µg/m³ of VOC.

[safakaltun]

Current “voc” sensor in z2m has a different unit that ppm. Twinguard uses ppm as voc unit in the bosch app. It also directly reports the voc ppm value.

Yup, Bosch is certainly doing some funny business here. Their own data sheet explicitly states, that the BME680 sensor is incapable of directly measuring CO2 levels. They absolutely have to do some calculations on their end, to produce a ppm value from just AQI.

result.co2 = msg.data['airpurity'] * 10.0 + 500.0; On that note, how'd you come up with this specific conversion formula?

Yes, but it is based on the sniffs and what is shown in the bosch app. The minimum value the bosch app reports is 500.

[burmistrzak]

Yes, but it is based on the sniffs and what is shown in the bosch app. The minimum value the bosch app reports is 500.

Ok, so what I thought. 😊 Lastly, would you mind sharing your reasoning behind gathering all sensor readings only from the manuSpecificBosch3 cluster, when respective default clusters are available? 👀

[burmistrzak]

Update on my experiment: I've quickly implemented a VOC sensor and it's working great so far! 🥳 Simply multiplying the AQI (i.e. airpurity) with 71.22, and returning VOC density as an additional result with bosch_twinguard_measurement is all that's required.

[safakaltun]

Yes, but it is based on the sniffs and what is shown in the bosch app. The minimum value the bosch app reports is 500.

Ok, so what I thought. 😊 Lastly, would you mind sharing your reasoning behind gathering all sensor readings only from the manuSpecificBosch3 cluster, when respective default clusters are available? 👀

That was a technical requirement as far as I remember. The device reports those values to that cluster and did not use the respective clusters. You should be able to look at the sniffs if you want to dig deeper in the original issue. :)

[burmistrzak]

That was a technical requirement as far as I remember. The device reports those values to that cluster and did not use the respective clusters.

Would have said maybe it's a change in firmware, but our devices report the same swBuildId... Regardless, I was able configure reporting for these default clusters, ~~and have been receiving data regularly. For that to work, I however had to comment out most of your work in bosch_twinguard_measurement, sorry. 😅~~

bosch_twinguard_measurements: {
        cluster: 'manuSpecificBosch3',
        type: ['attributeReport', 'readResponse'],
        convert: (model, msg, publish, options, meta) => {
            const result = {};
//            if (msg.data.hasOwnProperty('humidity')) {
//                result.humidity = msg.data['humidity'] / 100.0;
//            }
            if (msg.data.hasOwnProperty('airpurity')) {
                result.co2 = msg.data['airpurity'] * 10.0 + 500.0;
                result.voc = msg.data['airpurity'] * 71.22;
            }
//            if (msg.data.hasOwnProperty('temperature')) {
//                result.temperature = msg.data['temperature'] / 100.0;
//            }
//            if (msg.data.hasOwnProperty('illuminance_lux')) {
//                result.illuminance_lux = msg.data['illuminance_lux'] / 2.0;
//            }
//            if (msg.data.hasOwnProperty('battery')) {
//                result.battery = msg.data['battery'] / 2.0;
//            }
            return result;
        },
    },

Certainly not finished, but the Twinguard is seemingly providing readings at the default clusters when configured like this:

            await reporting.bind(device.getEndpoint(4), coordinatorEndpointB, ['genPowerCfg']);
            await reporting.bind(device.getEndpoint(5), coordinatorEndpointB, ['msTemperatureMeasurement']);
            await reporting.bind(device.getEndpoint(9), coordinatorEndpointB, ['msRelativeHumidity']);
            await device.getEndpoint(4).configureReporting('genPowerCfg', [{
                attribute: 'batteryPercentageRemaining',
                minimumReportInterval: constants.repInterval.HOUR,
                maximumReportInterval: 7200,
                reportableChange: 0
            }]);
            await device.getEndpoint(5).configureReporting('msTemperatureMeasurement', [{
                attribute: 'measuredValue',
                minimumReportInterval: 30,
                maximumReportInterval: constants.repInterval.MINUTES_5,
                reportableChange: 20
            }]);
            await device.getEndpoint(9).configureReporting('msRelativeHumidity', [{
                attribute: 'measuredValue',
                minimumReportInterval: 30,
                maximumReportInterval: constants.repInterval.MINUTES_5,
                reportableChange: 100
            }]);

You should be able to look at the sniffs if you want to dig deeper in the original issue. :)

Yea, I used your PCAPs to do my initial investigations, before I had my own Twinguard paired, but somehow wasn't able to decrypt some of your sniffs. Missing Transport Key, I guess..?

Edit: Huh..? After restarting the Twinguard, reporting no longer works. No sure, what's going on here. @safakaltun I guess your approach in this case was correct after all! 😉 AFAICT, the default clusters are only updated when explicitly read...

[burmistrzak]

I have a hunch that unknown5 from the manuSpecificBosch3 cluster is (air) pressure in hPa. 💨

Can someone please check on their Twinguard? A weather station or similar should show a comparable value.

[burmistrzak]

Did some further reading, and it seems the IAQ (indoor air quality) index used by the Twinguard isn't actually a linear scale. I was also able to find the article mentioned in the BME680 datasheet's footnotes.

That scale only goes from 1–5, meaning an AQI of 250, and not 351, should actually be equal to 25000 µg/m³ of TVOC. While the difference between each IAQ level (1–5) is constant at 50, the equivalent amount of TVOC is increasing in a non-linear fashion.

Scaling factor for each IAQ level:

1. (0–50) * 6
2. (51–100) * 10
3. (101–150) * 20
4. (151–200) * 50
5. (201–250) * 100
6. (251–350) ? ☠️
7. (>351) ? ☠️

@Koenkk I've implemented a quick PoC, but I first need to make sure, the TVOC values roughly reflect the reported IAQ levels. Additional PR incoming. 😉

[burmistrzak]

Now that estimated TVOC levels are roughly correct, CO2 seems a bit too high in comparison... I agree with @safakaltun, that the lower bound for the BME680 is 500 ppm, but I believe the CO2/IAQ scale from there on is not linear. Guess I'll have to find a reliable chart for that one too... 😅