MH-Z19B ABC wont turn on when ABC as been turned off Wemos D1 mini

onsmam commented 4 years ago

Steps already tried...

[X] Tried a clean install (empty .bin files are included in the ZIP)
[X] Tested previous/other build (mention which one already tested)
[X] Tested on other node to make sure hardware isn't defective.
[ ] Verified if the problem is limited to a single plugin/controller

Summarize of the problem/feature request

When the ABC is turned off, i cant switch on the ABC anymore. the U value stays at 0. when i turn the ABC off in the log there is a message ABC disabled. but when i turn it on again, no message saying its been turned on.

i've compared the CO2 value against an netatmo, the grafical interface shows about the same graph. only about 200ppm to much.

Expected behavior

U value going up again, and the co2 output becoming the same as netatmo

Steps to reproduce

turn off ABC on a MH-Z19B
turn on ABC

yes hard and soft reset

System configuration

Hardware: Wemos D1 mini MH-Z19b

ESP Easy version: ESP_Easy_mega_20200812_normal_ESP8266_4M1M.bin

ESP Easy settings/screenshots:

Rules or log data

1179967:  Webserver args: 0: 'index' length: 1 1: 'page' length: 1
1186843:  Webserver args: 0: 'index' length: 1 1: 'page' length: 1 2: 'TDNUM' length: 2 3: 'TDN' length: 9 4: 'TDE' length: 2 5: 'taskde
1186845: SaveToFile: free stack: 3104
1186877: FILE : Saved config.dat
1186878: SaveToFile: free stack after: 3104
1186884: SaveToFile: free stack: 3360
1186915: FILE : Saved config.dat
1186916: SaveToFile: free stack after: 3360
1186917: SaveToFile: free stack: 3328
1186989: FILE : Saved config.dat
1186990: SaveToFile: free stack after: 3328
1186991: SaveToFile: free stack: 2992
1187003: FILE : Saved security.dat
1187004: SaveToFile: free stack after: 2992
1187013: MHZ19: Init OK 
1189692: WD   : Uptime 20 ConnectFailures 0 FreeMem 21504 WiFiStatus WL_CONNECTED
**1202047: MHZ19: Sent sensor ABC Disable!**
1202048: MHZ19: PPM value: 676 Temp/S/U values: 27/0/0.00
1212247:  Webserver args: 0: 'index' length: 1 1: 'page' length: 1 2: 'TDNUM' length: 2 3: 'TDN' length: 9 4: 'TDE' length: 2 5: 'taskde
1212249: SaveToFile: free stack: 3040
1212282: FILE : Saved config.dat
1212283: SaveToFile: free stack after: 3040
1212288: SaveToFile: free stack: 3296
1212320: FILE : Saved config.dat
1212320: SaveToFile: free stack after: 3296
1212322: SaveToFile: free stack: 3264
1212393: FILE : Saved config.dat
1212394: SaveToFile: free stack after: 3264
1212395: SaveToFile: free stack: 2928
1212407: FILE : Saved security.dat
1212408: SaveToFile: free stack after: 2928
1212416: MHZ19: Init OK 
1219692: WD   : Uptime 20 ConnectFailures 0 FreeMem 21440 WiFiStatus WL_CONNECTED
1227440: MHZ19: PPM value: 674 Temp/S/U values: 27/0/0.00

TD-er commented 4 years ago

First some information on how the ABC operates on this sensor. The ABC in the MH-Z19 is collecting the minimal internal RAW value of the sensor over a period of 24h. So if it is not running for over 24h with ABC enabled, it will not adjust the 400 ppm level.

I do have those running here for about 3 years now and it seems like they do age, which also can be clearly seen in the ABC jumping around. In this image, you can see the CO2 as reported by this MH-Z19, with ABC enabled. In this period, I always have the windows upen for at least a few hours a day so it should be able to get a good ABC level. Also I am in this room for a few hours a day, so the level should rise to 800 - 1200 ppm (depending on the number of hours working) As can be seen in the chart, the ABC had a hard time getting into the correct range again. N.B. this sensor has an uptime of over 60 days, so it was not power cycled in this period, or in the days before.

That being said, it looks like the plugin can't seem to find the sensor as "detected" is left empty. So maybe you could try to set ABC to "normal", save it and then power off the unit for a while (longer than a few seconds) to make sure all capacitors are discharged. It looks like the sensor is crashed, or at least entered some limbo state from where it cannot get out again.

As soon as the sensor is working again (either with ABC enabled or not), can you have a look at the checksum statistics after it is running for a while? I wonder if there are lots of fail/reset counts. If there are, then the communication to the sensor may be less reliable and then it is hard to say what has been received by the sensor when turning the ABC on or off.

onsmam commented 4 years ago

when i went on vacation i removed the sensor from the wall outlet. for 7 days. should be enough to drain the capacitors :-). after that the U value (wich i belief to be the ABC value) isnt rising. the sensor is working fine as you can see in this screenshot: the yellow line is from the netatmo, the brown one is from the MH-Z19b

there is no ABC correction visable in the graph, as you can see in the graph (7day period) the line is about 200 ppm off all the time. so i think this has something to do with the ABC, i can perform the baseline calibration but i saw in other posts that this is not the way to go.

before i replaced the bin (in order to solve it) the sensor was running for about 7 days without an failed reading

TD-er commented 4 years ago

The meaning of the U value is a bit unclear as it isn't documented. But I too believe it has to do with the raw internal value of the sensor.

N.B. the A and B version of the sensor differ in the output of S and U value as the B value at least doesn't show the "S" value anymore.

Judging on the chart, I notice your sensor has quite a big offset and also the absolute differences are lower compared with the other sensor. When only the ABC is off, then you would expect the values to scale accordingly and this seems to be a scaling in a different direction to what I would expect. So maybe not only the ABC is off, but also the 0 point. Did you once try to set the 0-point calibration? Or perhaps change the range of the sensor (2000 to 5000 or the other way around) ? Is the sensor sometimes receiving full IR light, like sunlight, during the day?

onsmam commented 4 years ago

i think it also has something to do with the update interval en persistence settings of both sensors maybe also the scaling/logarithmic settings of the graph, the peaks of 26-08 between 0800 and 1600h still the offset is about 200 ppm, sometimes more than others but again, i think this as something to do with de update interval, als the esp will send every update of the sensor. Netatmo as interval of 10 min or so.

at one day i thought the ABC was ok, en i switched to disable the ABC, since then there is an offset of about 200 ppm.

for now i wil apply a calculation rule to the value with -200 will check how that will turn out.

never dit a change of range, nor an 0-point calibration. i wanted to do that, until i came in to the warnings not to do that, so i didn't. also the sensor is not receiving any direct light during the day.

so i really think the ABC stays off. maybe thats the ali-express crap... about the lowest value; we have WTW (don't know the english word). at night the fan is at high so there is fresh air pumped in the house, so i guess the lowest value during the night/day will be the outside value, so i think the ABC will work (at my place)

so then there is the question, why did i turn it off :-)

i had read on the forums that this will not work. i didn't test just switched it off when the value seemed to be ok..

TD-er commented 4 years ago

Well the sensor does seem to get somewhat saturated ("verzadigd" :) ) when it ages. This seems to result in lower output values for the same CO2 concentration. The firmware in the sensor does clip the values to 400 ppm, so you should at some point start to see straight horizontal lines at 400 ppm as it actually measures a value that would be below 400 ppm, when using the old calibration data. The ABC does help to mitigate this a bit, by setting a new value for what is considered to be a 400 ppm value and thus scaling the output values accordingly.

If the sensor is aging, the raw values of 2 calibration points may get closer to each other, which could result in quite significant jumps when updating the ABC.

Updating the 0-point calibration is something you cannot do on your own as it will be next to impossible to do it right. Also forcing the "400 ppm" value is very likely not accurate, as the outside air is for sure not always 400 ppm. For example the outside air near my parked car sometimes reaches 800 ppm during the night as I live in the country side, with a lot of vegitation around. These plants "exhail" CO2 during the night.

I don't know what is the best for these MH-Z19 sensors, leave the ABC on or off. With ABC off, you will eventually see output values being too low (and lots of samples at "400ppm") as the sensor ages. With ABC on, you will see quite a bit offset every now and then when ventilation of the room wasn't optimal for 24h. (e.g. only ventilating during the night or living in a city)

ksga commented 3 years ago

I have the same issue. Same output in the log as @onsmam when enabling/disabling ABC. The sensor has been running uninterrupted for roughly six days (fairly small office with window partly open all the time), and still the baseline has not been reset (lowest values 850-900PPM). CO2 As you can clearly see I decided to close the window yesterday afternoon (autumn is coming), and as a result the lowest value was far higher at ~1600PPM. I would imagine that I would run ABC during summer, when the window is basically always open (room faces south), and disable it during the winter when full ventilation is not achieved regularly.

TD-er commented 3 years ago

Maybe it is possible to automate the ABC on/off for this sensor. For example turn on on Friday 12h and turn off on Monday morning. This way the weekend can be used to reset the 400 ppm level.

The 24h window of the MH-Z19 is just not a practical value.

ksga commented 3 years ago

Maybe it is possible to automate the ABC on/off for this sensor. For example turn on on Friday 12h and turn off on Monday morning. This way the weekend can be used to reset the 400 ppm level.

The 24h window of the MH-Z19 is just not a practical value.

Maybe - primary issue though - getting ABC to work on the device!

onsmam commented 3 years ago

Maybe it is possible to automate the ABC on/off for this sensor. For example turn on on Friday 12h and turn off on Monday morning. This way the weekend can be used to reset the 400 ppm level. The 24h window of the MH-Z19 is just not a practical value.

Maybe - primary issue though - getting ABC to work on the device!

just what i was about to say :-)

i think the problem is that the ABC wont turn on again..

i have to little coding skills to debug. if some can give me troubleshooting steps im glad to help.

TD-er commented 3 years ago

If the offset has changed, then the ABC is on.

What happens with the ABC off, is that over time the peak CO2 values are lower under the same conditions and the device will show 400 ppm (or 403) over a longer time during the day.

If the lowest recorded value during the day jumps around, then ABC is on.

ksga commented 3 years ago

If the offset has changed, then the ABC is on.

What happens with the ABC off, is that over time the peak CO2 values are lower under the same conditions and the device will show 400 ppm (or 403) over a longer time during the day.

If the lowest recorded value during the day jumps around, then ABC is on.

Exactly - but I think what happens for @onsmam, and myself, is that even though ABC is on in the device config, it doesn't perform the calibration, and offset remains. So - is there any way we can force ABC on - perhaps connection directly to the device with a serial/USB connector and sending some command from a terminal? As is shown in the log output in the top post, we get a confirmation when disabling ABC, but none when re-enabling it - and evidence suggest it is off :(

ksga commented 3 years ago

BTW - I ran the sensor next to a sensor in a large ventilation unit some time ago, and the offset was stable at around 500PPM, so the sensor itself seems to perform as expected apart from no ABC

TD-er commented 3 years ago

I just looked at the code of the sensor. Do you use filtering? If you don't use filtering, then the ABC is probably also not turned on again. (seems like a bug to me)

There are commands available for the plugin, so as a test you can run the commands to see if those have an effect.

mhzabcenable
mhzabcdisable

Those were the commands I was thinking of while typing my suggestion of turning it on during the Weekend and off on Monday.

ksga commented 3 years ago

I just looked at the code of the sensor. Do you use filtering? If you don't use filtering, then the ABC is probably also not turned on again. (seems like a bug to me)

There are commands available for the plugin, so as a test you can run the commands to see if those have an effect.
* `mhzabcenable`

* `mhzabcdisable`
Those were the commands I was thinking of while typing my suggestion of turning it on during the Weekend and off on Monday.

Ran the commands: 514510048: Command: mhzabcdisable 514510058: MHZ19: Sent sensor ABC Disable! 514515188: Webserver args: 0: 'cmd' length: 12 514554232: Command: mhzabcenable 514554242: MHZ19: Sent sensor ABC Enable! 514559388: MHZ19: Received command acknowledgment! Will see if it actually makes a jump within the next 24h period :)

onsmam commented 3 years ago

indeed, also when i send the mhzabcenable via the tools i get Command: mhzabcenable, MHZ19: Sent sensor ABC Enable!, MHZ19: Received command acknowledgment! so when you switch to normal in the device config this command is not send.. thats what i was trying to say... how can we fix esp easy, my coding skills are slim/non existing...

TD-er commented 3 years ago

Well now we know where to look, it does seem like a simple fix.

Will have a look.

ksga commented 3 years ago

Only one problem remain for me now - no ABC has happened yet, and 24h has definitely passed :( Well ventilated baseline still roughly 850-900PPM...

TD-er commented 3 years ago

24h since you sent the ABC enabled you mean?

I have no clue what the sensor uses as reference for when it starts the cyclus. For example, what if it always does keep an ABC periodic cycle, but just doesn't collect the actual lowest value and also doesn't apply it when disabled. This then means it starts a new cycle and then may look at the setting to see if it needs to actively monitor the lowest recorded value.

So please wait for upto 48h after sending the command to see if it starts showing a jump in data to correct the 400 ppm level.

ksga commented 3 years ago

Will do 😊

ksga commented 3 years ago

24h since you sent the ABC enabled you mean?

I have no clue what the sensor uses as reference for when it starts the cyclus. For example, what if it always does keep an ABC periodic cycle, but just doesn't collect the actual lowest value and also doesn't apply it when disabled. This then means it starts a new cycle and then may look at the setting to see if it needs to actively monitor the lowest recorded value.

So please wait for upto 48h after sending the command to see if it starts showing a jump in data to correct the 400 ppm level.

Still no base calibration done... No useful debug information either :(

TD-er commented 3 years ago

Should I make a test build for this PR? https://github.com/letscontrolit/ESPEasy/pull/3302

ksga commented 3 years ago

Should I make a test build for this PR?

3302

That would be awesome. Running it on a Wemos D1

TD-er commented 3 years ago

A test build:

ksga commented 3 years ago

A test build:

* [ESPEasy_ESP32_mega-20201009-5-PR_3302.zip](https://www.dropbox.com/s/g2x9r863fl05pq3/ESPEasy_ESP32_mega-20201009-5-PR_3302.zip?dl=0)

* [ESPEasy_ESP82xx_mega-20201009-5-PR_3302.zip](https://www.dropbox.com/s/piqiphlij19qbnh/ESPEasy_ESP82xx_mega-20201009-5-PR_3302.zip?dl=0)

Had some WiFi issues (2.4GHz radio died and magically resurrected after 15h or so???) - but test build flashed (factory reset, mhzabcdisable + mhzabcenable just to make sure nothing would interfere) and waiting to see if anything else happens :D

TD-er commented 3 years ago

Yeah I also have seen some strange issues with WiFi in the last 2 weeks.

I guess there is something fishy with the (new) timers used in WiFi code. For example it takes 10+ minutes for a unit to try the 2nd WiFi credentials while that should be 10 - 20 seconds max. Maybe your 15h thingy is something similar (perhaps a factor 1000 error somewhere as the new timers now use uint64 in usec)

ksga commented 3 years ago

Sorry - wasn't clear enough... It was my router that had a stroke of some kind... But I've noticed that the test build seems to have quite poor performance.

Pages load at a very uneven pace, and quite a few timeouts on the log page (>> NetworkError when attempting to fetch resource. <<)- no connectionfailures though, so just seems to be very laggy. Didn't have that issue beforehand, and now tried with the ESP_Easy_mega_20201011_normal_alt_wifi_ESP8266_4M1M build with same poor performance

TD-er commented 3 years ago

Can you test with a build from end of August

TD-er commented 3 years ago

New test build of this PR, including latest WiFi fixes.

ksga commented 3 years ago

Running the first test build for more than four days without reboot - and no ABC happening. Will try the new build and report back.

ksga commented 3 years ago

Some initial log output, with a few comments, from newest test build: https://pastebin.com/fPWar6p4

TD-er commented 3 years ago

Hmm let's hope you've been breathing straight into the sensor. Otherwise it might be a good idea to start ventilating the room as it is around 1700 ppm. (at least that's what is reported by the sensor)

ksga commented 3 years ago

Hmm let's hope you've been breathing straight into the sensor. Otherwise it might be a good idea to start ventilating the room as it is around 1700 ppm. (at least that's what is reported by the sensor)

Unfortunately not - my homeoffice is just very small - and no ventilation unless the window is open - and autumn is really coming in now, so open window equals cold feet ;) But when I leave the room for an hour or so with the window open, the sensor still drops to 850 or so, maybe a bit lower, but nowhere near 400

ksga commented 3 years ago

ConnectFailures 0 FreeMem 20672 WiFiStatus WL_CONNECTED makes me think that it might be the laptop I'm using that has trouble keeping the log flowing, and not the wifi performance of the ESP... We can probably just ignore that bit from now on.

TD-er commented 3 years ago

But when I leave the room for an hour or so with the window open, the sensor still drops to 850 or so, maybe a bit lower, but nowhere near 400

Well maybe you should turn the ABC on... on wait... ;)

onsmam commented 3 years ago

indeed, also when i send the mhzabcenable via the tools i get Command: mhzabcenable, MHZ19: Sent sensor ABC Enable!, MHZ19: Received command acknowledgment! so when you switch to normal in the device config this command is not send.. thats what i was trying to say... how can we fix esp easy, my coding skills are slim/non existing...

after 10 days i dont see a major fall or rise in the graph.. the only thing i see is that in the beginning de lowest value is 400 and is stable for quite some time... the time it is at the lowest value is decreasing... so i think it is working.... im not sure... by the way i still havent installed the test build.

TD-er commented 3 years ago

This chart does indeed show the sensor did apply a small base line correction and does now seem to have a proper calibrated 400 ppm level. At first it was indeed sometimes "clipping" at 400 ppm, indicating the measured value appeared to be below 400 ppm with the then active calibration. Since the 16th this does no longer occur, indicating the base line is now OK as the lowest value still touches 400 ppm. The ABC adjustments were probably small, so you won't notice them other than the straight lines at 400 ppm.

Just as an illustration the SenseAir S8 sensor mounted in my car. This one somehow got damaged about 6 weeks ago and suddenly altered the 400 ppm ABC value to 900 ppm. As it uses a 7 day ABC interval and only allows to use steps of 25% of the required stepsize, it still hasn't returned to the 400 ppm level.

As this sensor was never intended to be used outside, let alone mounted in a car, I don't blame the sensor :) It is mounted right behind my gril, in front of the radiator. So it endured 20'000+ km of vibrations, winter cold, summer heat and even driving at max. speed of my car on the German Autobahn. (193 km/h measured on GPS, not too bad for a 20 year old diesel with 622'000 km)

letscontrolit / ESPEasy