Garbage characters randomly received by HardwareSerial when communicating with SIM800L modem

[toomasz]

Hardware:

Board: TTGO Bluetooth&Battery Core Installation/update date: 15 dec 2018 IDE name: Visual Micro Flash Frequency: 80Mhz PSRAM enabled: no Upload Speed: 115200 Computer OS: Windows 10

Description:

Hi, I'm designing a library for Simcom gsm modems on ESP32. Since the beginning im fighting with strange serial behavior - which is that HardwareSerial sometimes receives garbage characters from GSM modem, especially when it's transmitting/receiving.

Example:

[GSM] => AT+CSQ
[GSM]  <= +CSQ: 11,0
[GSM]  <= OK
[GSM] --- 
[GSM] => AT+CBC
[GSM]  <= +CBC: 0,91,4132
[GSM]  <= OK
[GSM] --- 
[GSM] => AT+CPIN?
[GSM]  <= +CPIN: READY
[GSM]  <= OK
[GSM]  <= ���������������n�������W����������������o{_��o��������s���o����������V��߿{���������������~������w�������������������߿w�o���������o��������~~����7�
[GSM]  <= RING
[GSM]  <= +CLIP: "+48XXXXXXXXX",145,"",0,"",0
[GSM] --- 
[GSM] => AT+CREG?
[GSM]  <= +CREG: 0,5
[GSM]  <= OK
[GSM] --- 
battery: 91%, voltage = 4.13 V
Signal quality: 11, operator = 
[GSM] => AT
[GSM]  <= }�����������?���������������}�������������~��������������_�����������������������-���v��l��{��w?���׾��������;�����w������O_��=���������w���j��u����?�����
[GSM]  <= OK
[GSM] --- 

Some facts:

• Problem occurs mainly when antenna is attached to SIM800L

  • I'm using ESP32 16 Pin for RX from gsm, 17 Pin for TX to modem
• I tried many different ESP32 boards and Sim800l modes and problem always accured.
• Problem happens mainly when I connect external antenna to SIM800L, without it most of the time there is no garbage on serial.

• I have proper power source that is capable of giving 2A for SIM800L and also external 1000uF capacitor attached to GND and VCC of module.

  • I didn't have this problem when I played with modem on regular Arduino UNO
  • Garbage characters occur when modem is transmitting/received packets so while making call, while registering to network, while sending/receiving data
  • Issue tracking the problem on my library: https://github.com/toomasz/SimcomGsmLib/issues/1
  • Serial initialization code: https://github.com/toomasz/SimcomGsmLib/blob/master/src/SimcomAtCommandsEsp32.h

  Questions:

• Does ESP32 have some kind of filtering for UART that can be activated?
• Why might ESP32 serial be more susectible to interference than Arduino one?

[copercini]

What baud rate are you using to communicate to Sim800l ?

[toomasz]

@copercini 115200, problem occurs on all baud rates though. Even on 9600.

[euquiq]

@toomasz I have a similar configuration ... I am using over 100 esp32 and SIM900L (although I used also SIM800L earlier on) and I have NOT experienced this behavior.

Frankly I would recommend you to check either your electrical connections / design. And even check board quality. Not less important, how are you powering up the whole thing. Which SIM800L board are you using (also, are you aware that SIM800L is an ANCIENT chip?) (Actually SIM900L is also quite old :) )

Also maybe you can post some code ... as in HOW do you begin your serial port ? And last and not least, which version of ESP32 ARDUINO libs are you using ? latest ?

Hope you can fix it !

[copercini]

I was testing this module with TinyGSM library yesterday, the problems were:

• some bugged characters when baud rate is 115200 or higher, then worked fine at 9600 or 4800,
• signal problems when power supply is weak

But nothing like your issue.... maybe you can try test with TinyGSM and see if the problem continues

[euquiq]

@copercini Tiny GSM library did not work for me. If I correctly recall, I posted the issue on their github page, and it turned out the library has problems inside esp32 architecture / tasks and memory handling (Again if I correctly recall, due to using Strings inside the lib).

A quick googleing throws:

https://github.com/vshymanskyy/TinyGSM/issues/133

[copercini]

@euquiq I didn't test it extensively, but was working here with this code https://gist.github.com/copercini/4a47ba2375a96320918c47ef6b7d3bbd

[euquiq]

@copercini In my case, I needed to use tinyGSM inside a task ... I think it was inside there that things went berzerk. It was a long time ago, so I cannot provide details. I really wanted it to work, tho, but I ended up doing everything my "myself" and I got it working.

[toomasz]

I managed to make it more stable by adding TTL lever converter between ESP32 and SIM800L since ESP32 is 2.2 to 3.3V and Sim800l is 3.4 to 4.4 V

Nevertheless sometimes i still get garbage characters, for example when GPRS connection is open and I try to call module number.

@euquiq Im using last ESP32 revision from github, actually I'm developing my library for simcom modules(it aims not to use Arduino strings and be non blocking) and you can see my initialization code here: https://github.com/toomasz/SimcomGsmLib/blob/master/src/SimcomAtCommandsEsp32.h My library has built in detection for garbage on serial so that why im able to reproduce it I think. Sometimes you might not even notice that with other libraries, you just get command timeout and garbage gets unnoticed.

What pins are you using for TX/RX on ESP32 side?

Yes, i do realize that its ancient and buggy module :) But I didn't have this problem with arduino so that why im reporting issue here.

Anyway I don't think problem is related to any library code, i was also getting strange characters when using basic serial proxy program.

[euquiq]

@toomasz, I am using hardwareserial 2 (GPIO17 and GPIO16):

#include "HardwareSerial.h" //CONNECTING INTO GSM MODEM

and on setup():

Serial2.begin(115200, SERIAL_8N1); //GSM MODEM CONNECTIVITY

[euquiq]

@toomasz I got a possible clue:

Do you have a SIM installed on that SIM800L ? Could it be that your telco provider is sending you PROVISIONING PACKETS ?? It did happen down here to me with ONE telco: They detect that you are not contacting the APN for -say- MMS pictures and they PUSH into your SIM800L a PACKET "asking" for autoconfiguration.

I recall that SUDDENLY, I had "GARBAGE" along with my stuff, and it was the incoming provisioning GSM PACKET or whatever it was called.

You will need to google it a bit, but it was kind of unavoidable, sudden, and binary. It would help if you could create a better "printout" of incoming data, not only showing ascii chars, but also de binary values, so you can see if the garbage has the correct "pattern" (and even google it)

[toomasz]

@euquiq I don't think SIM800L would send provisioning packets as random garbage characters to SIM800L. These random characters are usually interrupting normal AT commands response so my library cannot parse it. Also, i have garbage especially when I attach external antenna to SIM800l so i'm almost sure that this problem is related to ESP32 uart sensitivity. Anyway, for now problem is almost gone when SIM800L is attached through logic level converter.

[me-no-dev]

Is it possible that a buffer termination is not added in some case and some garbage (super secret data) is printed out until 0 is reached? I did not look at the library code...

[euquiq]

@toomasz I will not insist, I will only say the following: the provisioning packets should be received by your SIM800L after being powered on, within a minute or two. and be periodic.

They will have priority, and in my case it was EXACTLY like that: They would go over AT commands or anything. My SIM800L (and after that same thing with the SIM900L just inserted those bytes whenever they arrived, into the serial stream. In all cases, if you would decode in bytes, that "garbage" you could eventually detect if there is a pattern, and even google it, or just dismiss it as garbage.

[lrmoreno007]

I have the same garbage with the A6 (and with SIM900) but I use SoftwareSerial and what I do is wait for response and retry 3 times:

Global:

#include <SoftwareSerial.h>   // https://github.com/plerup/espsoftwareserial
boolean response_SIM900 = false; 
char PIN[] = "0787";
SoftwareSerial SIM900(D5, D6, false, 1024); // Pin de RXD, Pin de TXD

Wait response function:

void waitFor(const char *resp1, const char *resp2) {
  unsigned long entry = millis();
  String reply = "";
  int timeout = 2000;
  do {
    reply += SIM900.readString();
    yield();
  }
  while (((reply.indexOf(resp1) + reply.indexOf(resp2)) == -2) && ((millis() - entry) < timeout));

  if ((millis() - entry) >= timeout) {
    Serial_Telnet_ln("ERROR: Tiempo de espera superado");
    response_SIM900 = false;
  }
  else if (reply.indexOf(resp1) + reply.indexOf(resp2) > -2) {
    if (reply.indexOf(resp1) != -1) {
      Serial_Telnet_ln("OK");
      response_SIM900 = true;
    }
    if (reply.indexOf(resp2) != -1) {
      Serial_Telnet("ERROR: ");
      Serial_Telnet_ln(reply.c_str());
      response_SIM900 = false;
    }
  }
  else {
    Serial_Telnet("ERROR: Desconocido");
    response_SIM900 = false;
  }
}

Setup:

void setup() {
  Serial.begin(115200);
  SIM900.begin(115200);
}

In loop (or other function called from loop):

    response_SIM900 = false;
    int retry = 3;
    while (!response_SIM900 && retry > 0) {
      SIM900.print("AT+CPIN="); //Introducimos el PIN (variable PIN)
      SIM900.println(PIN);
      SIM900.flush();
      waitFor("OK", "+CME");
      delay(3000);
      retry--;
    }

Excuse me: Serial_Telnet_ln and Serial_Telnet are my functions that send message to Telnet and Serial at same time.

[lrmoreno007]

You can also activate the response in "TA format" with ATV0, but I don't know if garbage appears then (I don't try it).

Or supress any response with ATQ1:

Manuals: SIM900 http://www.jechavarria.com/wp-content/uploads/2015/05/SIM900-AT-Commands-Manual_V1.07.pdf A6 https://www.makerfabs.com/desfile/files/A6A7A6CA20_AT_Commends.pdf

On the other hand I think garbage are interferences.

[toomasz]

@lrmoreno007 SoftwareSerial as far as I tested it, didn't provide stable communication. I also got garbage from it but different kind of it :) And it was almost gone when baud rate changed was changed from higher one to 9600. I also think this is interference.

I'm just wondering why ESP32 UART is so susectible to it compared to ATMegas and if it can be avoided in any way, especially software way on esp-idf side.

Retrying in my case is no-go. I cannot retry command like CIPSEND as it would broke consistency of data stream on tcp connection.

[lrmoreno007]

I'm just wondering why ESP32 UART is so susectible to it compared to ATMegas and if it can be avoided in any way, especially software way on esp-idf side.

The first reason that comes to mind is that ATMega works at 5V and ESPs work at 3.3v. Less voltage means more sensitive to interference.

I forgot to say that I am using an ESP8266, not an ESP32

[me-no-dev]

are you sure that the data is really corrupted or do you judge by the garbage in the serial terminal?

[toomasz]

@me-no-dev I was judging by garbage in serial terminal and timeouts due to failure to parse response but now I also have detection of garbage built in into library: https://github.com/toomasz/SimcomGsmLib/blob/b46a5fc9756ab400554c89d08cd0203505cb4bc0/src/Parsing/ParsingHelpers.cpp#L165

[toomasz]

@euquiq Im still having problem with garbage, even after adding TTL lever converter. Would you mind sharing your setup? What ESP32 board are you using, which pins, which SIM900l board, etc. Thanks a lot.

[toomasz]

Ok, I guess I have finally solved my problem by using putting double ferrite choke on RX line. Please reopen if needed.

[imranaalam]

@euquiq can you please share the setup and loop running code. i have been trying for last 5 days and getting garbage on the sim800L modem through connecting usb to UART.

[toomasz]

@euquiq You can solve this problem by two ways: Put ferrite choke on sim800 tx -> esp32 rx Make sim800 tx -> esp32 rx connecion short, I'm not getting any garbage anymore where connection is around 4 millimeters

[Jeroen88]

@toomasz Could you specify what ferrite choke you are using? Just put it into the SIMCOM tx to ESP32 Rx line?

[arsz6733]

@toomasz Hi Toomasz. I've you issue in most of my designs and I haven't find any solution. I experience this issue when I wanted to to send data(in burst mode) and also sometimes when getting signal from BTS ! and in my experience all the problems have something with antenna placement!

Would you please tell how you find out that it's good to place a ferrite choke on sim800 tx? I also have problems in my ethernet connection during burst mode

[Jeroen88]

@arsz6733 maybe you ca try to put a 10k pulldown resistor between ESP32 Rx and ground?_

[arsz6733]

@Jeroen88 actully it's on an already built pcb and I can't add that to it. but as I know Uart tx and rx could be pullup not pulldown

[Jeroen88]

Well maybe you could try a pull up then. Just take a THT resistor and solder the wires to the pcb to find out if it works. If it does redesign your pcb

[arsz6733]

@Jeroen88 I tried this before and it didn't help in any way

[Jeroen88]

@arsz6733 I had similar problems with a sim7020 board and added a pull down to Rx. The pull down helped me to get rid of the problems. Later I realized that the SIM7020 uses 1.8v logic and I connected it directly to the ESP32 pins, so 3v3. My board has a level shifter build in, so I redesigned my PCB to use the level shifted pins. This also helped to get rid of the random characters.

Because of your remark that a Rx may have a pull up I checked this document, page 23. Indeed you are right, this document also uses a pull up.

My PCB also has the back side almost completely covered in a layer of copper connected to the ground for shielding.

It might be worth the effort to try the reference design of the document in the link. It may work for the sim800 also. I believe the problem is caused by noise picked up bij the ESP32 on the Rx line, possibly generated by the SIM radio. I do not believe the random characters are send by the SIM through the uart.

[toomasz]

@arsz6733 I was still getting problems with ferrite chokes after all - especially when network signal was weak and SIM800 was transmitting with high power. So my final solution was to place sim800 as close as possible to esp32 so that TX and RX cables are short.

[AyushMarsian]

@arsz6733 I am using SIM800L with ESP32 on UART2, was facing the same issue.

However, I found the solution by covering the TxD and RxD wire by shield of metal wire and ground the shield. Keep TxD and RxD lines far from antenna section. I will suggest, if you are designing PCB the try to apply ground pouring beside the track. This will help you if your GSM module is far from controller.

[sajidshaikh123]

Put 1K resistor in between SIM800 TX pin and ESP RX pin

[DhruvaG2000]

However, I find it unexplained that the same simcom with same wire length and connections works perfectly fine with arduino UNO, but gives so many issues with ESP32?? Is the 3v3 voltage level of ESP the reason behind this or something else??

[mamadgm]

I have The Same Garbage Characters With ESP32 but Its Fine With Arduino Uno i am About To buy some ferrite chokes.Hope That will solve the issue

[mamadgm]

I Finaly solve my problem by using 2 10K resistor on both rx and tx of both ESP32 And Sim800L And Im so happy I realy suggest you to do that cause its now Perfect NICE

[luconedj]

Hi all, try to revitalise this old post because years go fast but shitty problems remains 🙃 I'm facing an even worst issue with not only garbage on UART but also Crash and reboot with ESP32S3 devkit... I tried ferrite choke (a big one from an old laptop power supply) wit no turn, with turn, near ESP, on TX RX, on power line, everywhere with no success.. Rx Tx are around 10 cm, and I notice that module/antenna positioning amplify this fu__k behaviour.. I'm not with custom PCB, just prototyping on matrix board. Issue is totally absent with esp32s2 mini board!!! This is driving me crazy because I need more power for my app and I cannot downgrade to s2mini.. Do you have any other idea? Thank you guys! Luca

[busyBeaver1]

I had a similar problem. The noise problem itself was solved by adding an rc-filter to the tx of sim800l (C4 and R12 on the image). Works perfectly fine with 115200 baud rate

However the thing was still buggy as shit and frequently rebooted or disconnected randomly. 1. I added an inductor to the sim800l power line 2. added capacitors to the power line 3. added a ferrite ring onto the wire to antenna 4. covered as much as i could in copper foil, always connecting it to GND. Somehow the most effective measure seems to be to find a better placement for antenna empirically

Also I'm 100% sure that (at least in my case) the problem is related to noise from antenna carried through the air, not though the power line. Because I tried powering sim800l separately from esp32 it was connected to and nothing changed at all