SoftwareSerial fails to deliver last byte of Nextion End-Of-Packet until more data received

[avillacis]

Hardware: ESP32 Dev Module (YUBOX Node), no PSRAM, Nextion screen on pins 25 (TX), 32 (RX) Software: Arduino IDE 1.8.16, Arduino-ESP32 2.0.1 (released 9 November 2021)

When attempting to use SoftwareSerial for communication with a Nextion touchscreen, the last byte of the response packet (both command responses and event packets) gets "stuck" inside the library, until more data is sent from Nextion to the ESP32, at which the missing byte appears preceding the new data.

The Nextion serial protocol is a binary packet exchange, in which most, if not all packets, consist of a frame terminated by the binary sequence 0xFF,0xFF,0xFF. Data transmission from the ESP32 to the Nextion screen proceeds normally with both the default HardwareSerial class, and the SoftwareSerial one. However, when attempting to receive data, HardwareSerial behaves normally and all data bytes are eventually and timely received, but SoftwareSerial delivers all but the last 0xFF of the data response. This makes the packet appear "incomplete", and screws up both command response parsing and event handling on this setup.

Consider the following sample sketch (with NEX_SOFTWARE_SERIAL macro to switch between HardwareSerial and SoftwareSerial):

#include <Arduino.h>

//#define NEX_SOFTWARE_SERIAL 1

#define NEX_BAUD 9600

#ifdef NEX_SOFTWARE_SERIAL
#include <SoftwareSerial.h>
SoftwareSerial nexSerial;
#else
#define nexSerial Serial1
#endif

#if CONFIG_IDF_TARGET_ESP32
// Serial transmission pins for ESP32
#define NEXTION_SERIAL_TX           25
#define NEXTION_SERIAL_RX           32

#elif CONFIG_IDF_TARGET_ESP32S2
// Serial transmission pins for ESP32-S2
#define NEXTION_SERIAL_TX           GPIO_NUM_5
#define NEXTION_SERIAL_RX           GPIO_NUM_7

#else
#error Control pins not defined for target board!

#endif

void consumeBytes(uint32_t msec_timeout = 500)
{
  bool timeout = false;
  Serial.print("RECV: ");
  do {
    while (nexSerial.available()) {
      int16_t c = nexSerial.read();
      Serial.printf("%02x ", c);
    }
    uint32_t t = millis();
    while (!nexSerial.available() && millis() - t < msec_timeout) {
      delay(10);
    }
    timeout = !nexSerial.available();
  } while (!timeout);
  Serial.println("[EOI]");
}

void sendCommand(const char * cmd)
{
  nexSerial.print(cmd);
  nexSerial.write(0xFF);
  nexSerial.write(0xFF);
  nexSerial.write(0xFF);
  Serial.printf("SENT: %s\r\n", cmd);
}

void setup() {
  delay(3000);
  Serial.begin(115200);

    // Initialize serial port for Nextion
#ifdef NEX_SOFTWARE_SERIAL
    Serial.printf("Nextion using SoftwareSerial, speed %u baud...\r\n", NEX_BAUD);
    //Serial.println("nexSerial.enableRxGPIOPullup(false);");
    //nexSerial.enableRxGPIOPullup(false);
#else
    Serial.printf("Nextion using HardwareSerial, speed %u baud...\r\n", NEX_BAUD);
#endif
    nexSerial.begin(NEX_BAUD,
#ifdef NEX_SOFTWARE_SERIAL
        SWSERIAL_8N1,
#else
        SERIAL_8N1,
#endif
        NEXTION_SERIAL_RX, NEXTION_SERIAL_TX
#ifdef NEX_SOFTWARE_SERIAL
        ,
        false, 512
#endif
    );
}

void loop() {
  consumeBytes();

  delay(100);
}

With HardwareSerial, I get the following output:

14:28:00.891 -> Nextion using HardwareSerial, speed 9600 baud...
14:28:00.891 -> RECV: [EOI]
14:28:01.487 -> RECV: [EOI]
14:28:02.082 -> RECV: [EOI]
14:28:02.678 -> RECV: [EOI]
14:28:03.274 -> RECV: [EOI]
14:28:03.903 -> RECV: [EOI]
14:28:04.499 -> RECV: [EOI]
14:28:05.095 -> RECV: [EOI]
14:28:05.691 -> RECV: [EOI]
14:28:06.287 -> RECV: 65 00 09 01 ff ff ff [EOI]
14:28:07.115 -> RECV: [EOI]
14:28:07.711 -> RECV: [EOI]
14:28:08.307 -> RECV: [EOI]
14:28:08.903 -> RECV: [EOI]
14:28:09.532 -> RECV: [EOI]
14:28:10.128 -> RECV: [EOI]
14:28:10.724 -> RECV: 65 00 09 01 ff ff ff [EOI]
14:28:11.585 -> RECV: [EOI]
14:28:12.181 -> RECV: [EOI]
14:28:12.776 -> RECV: [EOI]
14:28:13.372 -> RECV: [EOI]
14:28:13.968 -> RECV: [EOI]
14:28:14.564 -> RECV: [EOI]
14:28:15.160 -> RECV: [EOI]
14:28:15.789 -> RECV: [EOI]
14:28:16.385 -> RECV: 65 00 09 01 ff ff ff [EOI]
14:28:17.081 -> RECV: [EOI]
14:28:17.676 -> RECV: [EOI]

With SoftwareSerial, I get this output instead:

14:24:46.879 -> Nextion using SoftwareSerial, speed 9600 baud...
14:24:46.879 -> RECV: [EOI]
14:24:47.475 -> RECV: [EOI]
14:24:48.071 -> RECV: [EOI]
14:24:48.668 -> RECV: [EOI]
14:24:49.264 -> RECV: [EOI]
14:24:49.861 -> RECV: [EOI]
14:24:50.457 -> RECV: [EOI]
14:24:51.086 -> RECV: [EOI]
14:24:51.682 -> RECV: [EOI]
14:24:52.278 -> RECV: 65 00 09 01 ff ff [EOI]
14:24:53.205 -> RECV: [EOI]
14:24:53.803 -> RECV: [EOI]
14:24:54.399 -> RECV: [EOI]
14:24:54.995 -> RECV: [EOI]
14:24:55.591 -> RECV: [EOI]
14:24:56.187 -> RECV: ff 65 00 09 01 ff ff [EOI]
14:24:56.948 -> RECV: [EOI]
14:24:57.544 -> RECV: [EOI]
14:24:58.141 -> RECV: [EOI]
14:24:58.737 -> RECV: [EOI]
14:24:59.366 -> RECV: [EOI]
14:24:59.962 -> RECV: [EOI]
14:25:00.558 -> RECV: [EOI]
14:25:01.155 -> RECV: ff 65 00 09 01 ff ff [EOI]
14:25:02.182 -> RECV: [EOI]
14:25:02.778 -> RECV: [EOI]
14:25:03.408 -> RECV: [EOI]

In both tests, a known-working firmware was used in the Nextion screen, which delivers the touch event packet, which consists of the byte sequence 65 00 09 01 ff ff ff .

With HardwareSerial, all bytes of the packet were received at the same time. With SoftwareSerial, however, the received sequence is 65 00 09 01 ff ff. The last 0xFF is missing at that time. However, when sending the next touch event packet, the ESP32 receives ff 65 00 09 01 ff ff. The missing byte from the previous data packet now appears preceding the new data packet.

[avillacis]

In file SoftwareSerial.cpp line 468 I see the following comment and code:

    // A stop bit can go undetected if leading data bits are at same level
    // and there was also no next start bit yet, so one word may be pending.
    // Check that there was no new ISR data received in the meantime, inserting an
    // extraneous stop level bit out of sequence breaks rx.
    if (m_rxCurBit > -1 && m_rxCurBit < m_pduBits - m_stopBits) {
        const uint32_t detectionCycles = (m_pduBits - m_stopBits - m_rxCurBit) * m_bitCycles;
        if (!m_isrBuffer->available() && ESP.getCycleCount() - m_isrLastCycle > detectionCycles) {
            // Produce faux stop bit level, prevents start bit maldetection
            // cycle's LSB is repurposed for the level bit
            rxBits(((m_isrLastCycle + detectionCycles) | 1) ^ m_invert);
        }
    }

At a wild guess, I believe there is a bug in this code section that prevents it from working properly.

[dok-net]

@avillacis Please verify that release 6.15.1, indeed fixes your issue. Thank you for pointing out this bug, it went previously unnoticed due to a mistake in the testing procedure.

[avillacis]

Confirmed fixed in ESP32. I can now read all bytes from Nextion as intended.