Winsen ZE25-O3 software serial skipping first octet

[lukas-kurka]

I'm trying to read O3 concentration from Winsen ZE25-O3 (datasheet). The sensor has two output modes - ACTIVE which sends out concentration message to serial every second and - QUESTION/ANSWER when it sends out the message only after receiving the request.

I have two HW configuration running the same code:

1) Arduino Leonardo (framework-arduino-avr 5.1.0, SoftwareSerial 1.0) 2) NodeMCU v1.0 (framework-arduinoespressif8266 2.7.4, EspSoftwareSerial 6.8.5)

On both I run the sketch below which uses the question/answer mode to retrieve the data from sensor.

THE PROBLEM : In QUESTION/ANSWER mode on Leonardo I get the message I'm expecting - FF8600140000001452 - which is 9 octets long, on NodeMCU I get only - 8600140000001452 - which is only 8 octets, the 1st octet "FF" is missing.

The communication towards the sensor is fine, it switches the modes when I send the correct payload to it (so it receives it in full). When I try the active upload mode I get the same correct output on both Leonardo and NodeMCU - FF8600140000001452 repeating every second. I have spent several evenings of testing and researching but so far haven't found what I'm doing wrong. At this moment my thoughts are that there is something wrong with EspSoftwareSerial (not itself, it maybe needs some more configuration from my side). One thought I came to - is it possible that EspSoftwareSerial is not ready to listen for the incoming data exactly the moment after sending out the last portion of outbound data?

With a little bit of coding I can of course parse the data buffer received using ACTIVE mode, however receiving data only when requested seems much more neat and elegant. For production it would probably be possible to use HW UART but for development I have it reserved/use it for debugging. But I'm so much curious to get Q&A mode up to work.

Code where the issue happens :

#include <Arduino.h>
#include <SoftwareSerial.h>

#if defined(ARDUINO_AVR_LEONARDO)
#define rxPin (8)
#define txPin (7)
#elif defined(ESP8266)
#define rxPin (12)
#define txPin (13)
#endif

// const uint8_t rxPin = 8;            // Arduino Leonardo
// const uint8_t txPin = 7;            // Arduino Leonardo
// const uint8_t rxPin = 12;            // NodeMCU
// const uint8_t txPin = 13;            // NodeMCU
const uint16_t baudRate = 9600;

const uint8_t MSG_LENGHT = 9;

const uint8_t comMsgQuestion[9] = {0xFF, 0x01, 0x78, 0x41, 0x00, 0x00, 0x00, 0x00, 0x46};     // payload that needs to be sent to sensor to switch to QUESTION/ANSWER mode
const uint8_t comMsgActive[9] = {0xFF, 0x01, 0x78, 0x40, 0x00, 0x00, 0x00, 0x00, 0x47};       // payload that needs to be sent to sensor to switch to ACTIVE UPLOAD mode - NOT USED
const uint8_t comMsgRequest[9] = {0xFF, 0x01, 0x86, 0x00, 0x00, 0x00, 0x00, 0x00, 0x79};      // payload that needs to be sent to sensor to REQUEST actual reading

SoftwareSerial o3serial(rxPin, txPin);

void setCommQuestionMode();
void requestSensorReading();
void getSensorReading();

// Configures sensor to question/answer mode so it sends the reading only when asked to do so.
void setCommQuestionMode(){
  for (uint8_t i = 0; i < MSG_LENGHT; i++){
    o3serial.write(comMsgQuestion[i]);
  }
}

// Requests the sensor to send reading to serial
void requestSensorReading(){
  for (uint8_t i = 0; i < MSG_LENGHT; i++){
    o3serial.write(comMsgRequest[i]);
  }
}

// Reads and stores raw data from serial port
void getSensorReading(){
  uint8_t sensorReadingSerial[MSG_LENGHT];  // mesage from the sensor is stored into this array

  requestSensorReading();                   // ask the sensor to send data
  delay(10);                                // wait a short while so the buffer can be filled

  // read the data from software serial buffer
  uint8_t counter=0;
  while (o3serial.available() > 0) {
    sensorReadingSerial[counter] = o3serial.read();             // read from the buffer
    if (sensorReadingSerial[counter] < 0xF) Serial.print("0");  // prinout leading zero if neccessary
    Serial.print(sensorReadingSerial[counter],HEX);             // printout to hardware serial for dubugging
    counter++;
  }
  Serial.println();
}

void setup() {
  // Initilize serial port for console ouptut and debugging
  Serial.begin(9600);
  while (!Serial){
  }

  // Initilize serial port for o3 sensor and configure sensor to question mode
  o3serial.begin(baudRate);
  o3serial.listen();
  setCommQuestionMode();
}

void loop() {
  getSensorReading();
  delay(1000);
}

Any idea what could be the problem?

Many thanks for the EspSoftwareSerial work!

[dok-net]

One first thought: could the

delay(10);                                // wait a short while so the buffer can be filled

be too short for the following

while (o3serial.available() > 0) {

such it simply skips the while block? Plus, your code seems to overrun the sensorSerialReading buffer quite easily.

[lukas-kurka]

delay(10);                                // wait a short while so the buffer can be filled

be too short for the following

while (o3serial.available() > 0) {

such it simply skips the while block?

That was one of my thougths as well - I've already tried to increase it, even significantly, but doesn't do the magic. Actually I would be surprised if it would, because if I would be reading input buffer too early I would expect to receive either nothing or only some of the first characters( and that the end of message would be stripped of). However what I'm missing in the reading is the 1st octet, the start byte - "FF" and the rest of message is successfully received. That is what's most weird to me.

I have tried to put in some timestamps, and I can confirm that it goes through the 'while' loop every time.

Plus, your code seems to overrun the sensorSerialReading buffer quite easily.

You mean that the array's lenght of 9 characters for reading out serial buffer is short? I haven't thought about that as the buffer is filled only upon my request and I know that every message is exactly 9 bytes long. But you are definitely right that it should be longer if something unexpected would pop out in the buffer by sensor error, noise etc. Anyway I have expanded that array and there are no signs of overflow.

Any other idea?

[dok-net]

Which version of ESP8266 Core for Arduino are you using? Can you please get the master branch from GitHub, uninstall any EspSoftwareSerial you may have installed via the Arduino library manager and follow the README instructions to make sure you are working with the latest EspSoftwareSerial, too? I may be completely fishing in the dark for you right now, and your issue doesn't exist in recent versions :-)

[lukas-kurka]

So far I had no issues so I'm using the latest stable release for today - ESP8266 Arduino core 2.7.4 and EspSoftwareSerial 6.8.5. As suggested I updated (at least I tried, only little experience of using git🤦‍♂️) to bleeding edge of Arduino-ESP8266 core however the issue persists..

> Executing task: platformio run <

Processing nodemcuv2 (platform: espressif8266; board: nodemcuv2; framework: arduino)
------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 (2.6.3) > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES: 
 - framework-arduinoespressif8266 3.0.0-dev+sha.fdc295d 
 - tool-esptool 1.413.0 (4.13) 
 - tool-esptoolpy 1.30000.201119 (3.0.0) 
 - toolchain-xtensa 5.100200.201223 (10.2.0)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 36 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <EspSoftwareSerial> 6.12.0
Building in release mode
Retrieving maximum program size .pio/build/nodemcuv2/firmware.elf
Checking size .pio/build/nodemcuv2/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [===       ]  34.8% (used 28476 bytes from 81920 bytes)
Flash: [===       ]  26.0% (used 271297 bytes from 1044464 bytes)
====================================================== [SUCCESS] Took 1.07 seconds ======================================================

Terminal will be reused by tasks, press any key to close it.

[dok-net]

OK, something ridiculous is going on:

// Connect D4 to D7

#include <SoftwareSerial.h>

SoftwareSerial swSerial(D7);

void setup()
{
    Serial.begin(9600);
    Serial1.begin(9600);
    swSerial.begin(9600);
    while (!Serial) {}
    while (!Serial1) {}
    delay(100);
    Serial.println("Sending from HW Serial1 (TX on D4) to EspSoftwareSerial (RX on D7)");
}

void loop()
{
    Serial1.write(255);
    delay(2001);

    int avail;
    if (0 < (avail = swSerial.available()))
    {
        for (int i = 0; i < avail; ++i) {
            Serial.print(swSerial.read());
            Serial.print(" ");
        }
    }
    else
    {
        Serial.println("\nDropped octet detected");
    }
}

This works for me, except the very first iteration gives a Dropped octet detected once:

Sending from HW Serial1 (TX on D4) to EspSoftwareSerial (RX on D7)

Dropped octet detected
255 255 255 255 255 255 255 255 255 255 255 255 255 255

But - @devyte, do you have an explanation? - if I change the delay(2001) by a mere millisecond to delay(2000):

Sending from HW Serial1 (TX on D4) to EspSoftwareSerial (RX on D7)

Dropped octet detected

Dropped octet detected

Dropped octet detected

Dropped octet detected

Dropped octet detected

Dropped octet detected
183 
Dropped octet detected
223 
Dropped octet detected
223 255 
Dropped octet detected
223 
Dropped octet detected

[dok-net]

I'm still baffled by the exact 2000ms interval that triggered the issue, but I've pushed a fix to the PR #205 for this issue. @lukas-kurka Can you please update your copy of hardware/esp8266com/esp8266/libraries/SoftwareSerial to that PR and re-test?

[lukas-kurka]

I'm sorry @dok-net I'm not experienced developer so a better (simpler) way to demonstrate the issue didn't come to my mind.

1) As you have asked if it could be specific hardware piece related - I have checked 4 different microcontrollers based on ESP8266 and 2 different sensors in different combinations. They all behave the same (stripping 1st octet) except one. On that piece the output looked like this

8600140000001452
8600140000001452
FF8600140000001452
FF8600140000001452
8600140000001452
FF8600140000001452
FF8600140000001452
FF8600140000001452
FF8600140000001452

when it finally stabilized and stopped dropping the first octet. Little confusing to me.

2) I have also played with the wait interval as you suggested, even tried to step up 1ms during each operation cycle. In my case no change at all.

3) I apologize for my unknowledge of git, I have to read something about it before I'm able to pull the above mentioned fix. So far I did it just through platformio.ini in VSCode.

lib_deps =
  plerup/EspSoftwareSerial @ ^6.12.0
platform_packages = 
  framework-arduinoespressif8266 @ https://github.com/esp8266/Arduino.git
  mcspr/toolchain-xtensa@5.100200.201223

I will come back to you when I succeed.

[dok-net]

@lukas-kurka As you can imagine, I do recommend learning to use git / GitHub. In the meantime, if you can locate the sources for EspSoftwareSerial 6.12.0 where they were placed by Platformio, which you apparently are using, you can replace them with those contained in this downloadable up-to-date ZIP archive: https://github.com/dok-net/espsoftwareserial/archive/refs/heads/issue204.zip . src/SoftwareSerial.h and src/SoftwareSerial.cpp will suffice for your purposes.

[lukas-kurka]

So I managed to fetch the proposed fix in the proper way from github. Unfortunately it didn't solve the problem of dropping first octet in communication with the sensor.

I took two ESP8266 one configured to send FF octets via hardware UART and the other to receive it via sw serial and I'm even unable to recreate the strange behaviour you achieved no matter of fix / no fix version of espsoftwareswerial.

It seems the problem lies somewhere in the exact combination ESP8266 + sensor + espsoftwareserial. To make sure of it I will try to hook up the sensor to hw uart of ESP8266. However I'm not sure how to log the results for debugging then, will have to think of something.

[dok-net]

Please, I don't use Platformio, I can't know what exactly you mean by "the proper way" - these are important variables in getting the latest sources to compile, instead of whatever Platformio downloads. I've been there too many times with issue reports based on false assumptions by users on how to stitch together ESP8266 Core for Arduino and EspSoftwareSerial releases, so don't take this personal. Without logs, and without confirmation that you've run exactly the sketch source code that I provided above, I can't derive any conclusions from your report, I am sorry. As far as I am concerned, based on your report, thank you again for bringing this up, I have found and also fixed a long-standing issue that before had stayed under the radar.

Please run the sketch above, connect the required pins only, and let me know if your result is exactly like this, if it's not, make double sure that you are actually feeding the updated library sources from the PR to the compiler:

Opening port
Port open
Sending from HW Serial1 (TX on D4) to EspSoftwareSerial (RX on D7)
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 

[lukas-kurka]

I'm sorry that was only my little happiness that I was able to source the desired branch using git and not replacing the source code somewhere in directory structure. I wanted to achieve this as I can't be sure when Platformio decides to fetch the code again and rewrite my manual changes.

Anyway.. my output of your sketch using EspSoftwareSerial @ 6.12.0

Sending from HW Serial1 (TX on D4) to EspSoftwareSerial (RX on D7)

Dropped octet detected
255 255 255 255 255 255 255 255 255

My output using EspSoftwareSerial @ 6.12.1+sha.b187646

Sending from HW Serial1 (TX on D4) to EspSoftwareSerial (RX on D7)
255 255 255 255 255 255 255 255 255 255 255 255 255 255

Probably unnecessary but above this I have double checked the downloaded source code at it matches yours, so I dare to say we are both referring to the same code.

[lukas-kurka]

So when I get back to the sensor reading. On EspSoftwareSerial @ 6.12.1+sha.b187646 I run your sketch modified as little as possible to read the sensor serial data :

//Connect sensor's TX to ESP D7 and sensor's RX to D6

#include <SoftwareSerial.h>

SoftwareSerial swSerial(D7,D6); // added D6 to be able to request sensor for reading

void setup()
{
        // removed Serial1 as here we read real sensor no emulation
    Serial.begin(9600);
    swSerial.begin(9600);
    while (!Serial) {}
    delay(100);
        Serial.println("Sending to sensor from EspSoftwareSerial (TX on D6) and receiving from sensor on EspSoftwareSerial (RX on D7)");
}

void loop()
{
    // write predefined message to sensor so it returns it's 9 byte message
    const uint8_t comMsgRequest[9] = {0xFF, 0x01, 0x86, 0x00, 0x00, 0x00, 0x00, 0x00, 0x79};
    for (int i = 0; i < 9; i++){
        swSerial.write(comMsgRequest[i]);
    }
    delay(2001); 

    int avail;
    if (0 < (avail = swSerial.available()))
    {
        for (int i = 0; i < avail; ++i) {
            Serial.print(swSerial.read(), HEX);
            Serial.print(" ");
        }
        Serial.println();
    }
    else
    {
        Serial.println("No data");
    }
}

and get following results on EspSoftwareSerial @ 6.12.1+sha.b187646 (it is every time different, there is no pattern, the 1st octet is missing randomly) :

Sending to sensor from EspSoftwareSerial (TX on D6) and receiving from sensor on EspSoftwareSerial (RX on D7)
86 0 14 0 0 0 14 52
FF 86 0 14 0 0 0 14 52
86 0 14 0 0 0 14 52
86 0 14 0 0 0 14 52
FF 86 0 14 0 0 0 14 52
86 0 14 0 0 0 14 52
86 0 14 0 0 0 14 52
FF 86 0 14 0 0 0 14 52
FF 86 0 14 0 0 0 14 52
FF 86 0 14 0 0 0 14 52
86 0 14 0 0 0 14 52
86 0 14 0 0 0 14 52
86 0 14 0 0 0 14 52
86 0 14 0 0 0 14 52
86 0 14 0 0 0 14 52

The sensor message always starts with FF so I can't determine if the problem is generally first octet or specifically "FF". I have't been able to achieve the message to contain FF in other position. The same sketch (only with modified pin numbers to 8, 9) gets consistent readings on Arduino Leonardo.

Sending to sensor from SoftwareSerial (TX on 8) and receiving from sensor on SoftwareSerial (RX on 9)
FF 86 0 14 0 0 0 14 52
FF 86 0 14 0 0 0 14 52
FF 86 0 14 0 0 0 14 52
FF 86 0 14 0 0 0 14 52
FF 86 0 14 0 0 0 14 52

[dok-net]

Then swap Serial from (RX, TX) to (D7, D8) in order to use the HW UART with your device, and let EspSoftwareSerial do the logging. Does this solve your problem? First test, then yet again adapt the code below to your needs:

// Connect D4 to D7

#include <SoftwareSerial.h>

SoftwareSerial swSerial(RX, TX);

void setup()
{
    Serial.begin(9600);
    Serial.swap();
    while (!Serial) {}
    Serial1.begin(9600);
    while (!Serial1) {}
    swSerial.begin(9600);
    delay(100);
    swSerial.println("Sending from HW Serial1 (TX on D4) to HW Serial (RX on D7, TX on D8, EspSoftwareSerial logs on TX)");
}

void loop()
{
    Serial1.write(0xFF);
    delay(2000);

    int avail;
    if (0 < (avail = Serial.available()))
    {
        for (int i = 0; i < avail; ++i) {
            swSerial.print((char)Serial.read(), HEX);
            swSerial.print(" ");
        }
    }
    else
    {
        swSerial.println("\nDropped octet detected");
    }
}

[dok-net]

It could also be an electrical matter, pull-up / no pull-up, idle voltage, etc. This is getting out of scope regarding the time I can spend on this. Please report your findings, do try the HW UART sketch above, and if you can, measure the line voltages. I will close this issue, merge the PR and pull-request the ESP8266 Arduino core once I have made sure my PM sensor still works fine.

[lukas-kurka]

It could also be an electrical matter, pull-up / no pull-up, idle voltage, etc. This is getting out of scope regarding the time I can spend on this. Please report your findings, do try the HW UART sketch above, and if you can, measure the line voltages. I will close this issue, merge the PR and pull-request the ESP8266 Arduino core once I have made sure my PM sensor still works fine.

I have successfully tested HW UART. However I have decided to rather use the other mode of the sensor, where it actively sends data, I capture them using SWSERIAL and parse them.

Will try to do the measurements when I have more time as well.

@dok-net very appreciate your help! Can I buy you beer somewhere e.g. Patreon?

[dok-net]

@lukas-kurka Too bad we are still in the dark concerning the 0xff at your messages' start. Anyway, I'll ask the ESP8266 team to merge this latest EspSoftwareSerial release.