JSON deserialisation of Hammingstream, can't get it to work

[TLS1000]

Hi, I'm using a mega as port externder for a ESP32 via UART. The idea is to send JSON files with IO-status and requests back and forth. Because the size of the JSON doc is larger than UART capacity, it needs to be sent in pieces. I've tested it with the readBufferingstream class and this works great (thank you by the way)! I wanted to add some errorchecking by trying the hammingstream-class. In my test setup I make some dummy JSON about 800bytes. I think the sending works ok (I see it on scope) using the "serializeJson(doc, eccSerial1);", but on the receiver side the "DeserializationError err = deserializeJson(doc, eccSerial1);" returns error.

sender (arduino mega)

#include <ArduinoJson.h>
#include <StreamUtils.h>
// Create a new Stream that buffers all writes to Serial
//ReadBufferingStream bufferedSerial{Serial1, 64};
HammingStream<7, 4> eccSerial1(Serial1);

void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  Serial1.begin(115200);

}

void loop() {
  // put your main code here, to run repeatedly:
  long timestamp = millis();
  int value = analogRead(1);
  DynamicJsonDocument doc(2000);
  doc["timestamp"] = timestamp;
  doc["value"] = value;
  JsonArray bulk=doc.createNestedArray("bulk");
  for(int i=0;i<50;i++){
    bulk.add(micros());
  }

  // Send the JSON document over the "link" serial port
  //serializeJson(doc, bufferedSerial);
  serializeJson(doc, eccSerial1);

  // Even if it looks like the bytes are extracted one by one, they are actually
  // read by chunks of 64 bytes and placed in a buffer.
  while (eccSerial1.available()) {
    Serial1.write(eccSerial1.read());
    //Serial1.write(bufferedSerial.read());
  }

  delay(2000);
}

receiver(ESP32):

#include <HardwareSerial.h>
#include <ArduinoJson.h>
#include <StreamUtils.h>
HardwareSerial serial1(1);

HammingStream<7, 4> eccSerial1(serial1);

void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  serial1.begin(115200,SERIAL_8N1, 15, 2);
  //pinMode(15,OUTPUT);
  //pinMode(2,OUTPUT);
}

void loop() {
 // Check if the other Arduino is transmitting
  char response[2000]; 
  if(eccSerial1.available())
  {
    // Allocate the JSON document
    // This one must be bigger than for the sender because it must store the strings
    DynamicJsonDocument doc(2000);

    // Read the JSON document from the "link" serial port
    DeserializationError err = deserializeJson(doc, eccSerial1);
    //DeserializationError err = deserializeJson(doc,bufferedSerial );

    if (err == DeserializationError::Ok) 
    {
      // Print the values
      // (we must use as<T>() to resolve the ambiguity)
      Serial.print("timestamp = ");
      Serial.println(doc["timestamp"].as<long>());
      Serial.print("value = ");
      Serial.println(doc["value"].as<int>());
      serializeJsonPretty(doc, Serial);
    } 
    else 
    {
      // Print error to the "debug" serial port
      Serial.print("deserializeJson() returned ");
      Serial.println(err.c_str());

      // Flush all bytes in the "link" serial port buffer
      //while (bufferedSerial.available() > 0)
        //bufferedSerial.read();
      while (eccSerial1.available() > 0)
        eccSerial1.read();
    }
  }
  //delay(1000);
}

[bblanchon]

Hi @TLS1000,

Please apologize for the exceptional delay in my response; I'm currently very busy, and I didn't find the time to get a look into this issue. I'll get back to you as soon as possible.

Best regards, Benoit

[TLS1000]

Hi Benoit, no problem. As for now I'll use the bufferingstream class. Let me know if you would like me to test something as I have the setup ready. I think using the json and hammingstream between two controllers would be a very good combo.

[bblanchon]

Hi @TLS1000,

I was going to reproduce your setup when I saw several problems in your programs.

First, in the sender, you write directly to Serial1 bypassing the Hamming coding:

 while (eccSerial1.available()) {
    Serial1.write(eccSerial1.read());
    //Serial1.write(bufferedSerial.read());
  }`

Indeed, the error correction can recover from a flipped bit, but not from inserted bytes. I don't know if it really is the cause here, because you should not receive anything through eccSerial1, but you'd better fix this anyway.

Secondly, from my experience, the baud rate used for logging should be significantly higher than the transmission link (at least x10); otherwise, the logging could slow down the program to the point where the transmission link drops bytes. This is a problem for the Hamming encoding because it cannot recover from drop bytes: sender and receiver must stay in sync, only bit flips can be corrected. I recommend that you slow down the link during your diagnostic and then increase it back for production when you don't need to log.

Lastly, from your message, I understand that ReadBufferingStream was needed to make the original program work. If that's the case, then you should preserve it: the hamming encoding should only by an addition to the buffer, not a replacement. You can attach the HammingStream to the ReadBufferingStream or the other way around, it should not matter. BTW, did you try with a shorter payload?

Also, ESP32 has a 3.3V UARL whereas the Mega has a 5V UART; are you using a level adapter. If not, the 5V input on the RX pin could damage your ESP32 and possibly increase the error ratio.

Best regards, Benoit

[TLS1000]

Hi Benoit,

I simplified the test of the hammingstream:

sender:

HammingStream<7, 4> eccSerial1(Serial1);
void loop() {
  // put your main code here, to run repeatedly:

  eccSerial1.write('a');

  delay(2000);
}

receiver:

HardwareSerial serial1(1);
HammingStream<7, 4> eccSerial1(serial1);
void loop() {
    while (eccSerial1.available() > 0)
        Serial.write(eccSerial1.read());

  //delay(1000);
}

I would expect to see 'a'-s printed to the receivers' monitor, but each loop it prints '61'

On hardware, I use level shifters, and also the bufferstream class works so I don't suspect level conversion issue.

[TLS1000]

Does the hammingstream class also chop the payload to be sent into managable pieces for eg the UART?

In fact, instead of using the hammingstream, I would prefer using the bufferingstream but with a CRC16 check. Wouldn't it be a good idea to enable this automatically inside the readbufferingstream class?