getNewMessage is taking too long

[topsecretgtr]

Hi,

First of all, thank you for the library.

Second, I had implemented your inlineKeyboard function to my sketch. I am building a temperature and humidity controller using wemos D1R2, BME280 sensor, I2C 1602 LCD and 2ch relays. Before I implemented CTBot, I tested my sketch and were able to pull data from the sensor every 1 second. However, after implementing, each iteration lasted about 4 seconds. I used your inlineKeyboard examples and it also show that each iteration lasted about 4 seconds. May I know what did I do wrong?

Thank you in advance

#include "CTBot.h"

#define LIGHT_ON_CALLBACK  "lightON"  // callback data sent when "LIGHT ON" button is pressed
#define LIGHT_OFF_CALLBACK "lightOFF" // callback data sent when "LIGHT OFF" button is pressed

CTBot myBot;
CTBotInlineKeyboard myKbd;  // custom inline keyboard object helper

String ssid = "";     // REPLACE mySSID WITH YOUR WIFI SSID
String pass = ""; // REPLACE myPassword YOUR WIFI PASSWORD, IF ANY
String token = "";   // REPLACE myToken WITH YOUR TELEGRAM BOT TOKEN
uint8_t led = D5;            // the onboard ESP8266 LED.    
                            // If you have a NodeMCU you can use the BUILTIN_LED pin
                            // (replace 2 with BUILTIN_LED) 
                            // ATTENTION: this led use inverted logic

void setup() {
  // initialize the Serial
  Serial.begin(115200);
  Serial.println("Starting TelegramBot...");

  // connect the ESP8266 to the desired access point
  myBot.wifiConnect(ssid, pass);

  // set the telegram bot token
  myBot.setTelegramToken(token);

  // check if all things are ok
  if (myBot.testConnection())
    Serial.println("\ntestConnection OK");
  else
    Serial.println("\ntestConnection NOK");

  // set the pin connected to the LED to act as output pin
  pinMode(led, OUTPUT);
  digitalWrite(led, HIGH); // turn off the led (inverted logic!)

  // inline keyboard customization
  // add a query button to the first row of the inline keyboard
  myKbd.addButton("LIGHT ON", LIGHT_ON_CALLBACK, CTBotKeyboardButtonQuery);
  // add another query button to the first row of the inline keyboard
  myKbd.addButton("LIGHT OFF", LIGHT_OFF_CALLBACK, CTBotKeyboardButtonQuery);
  // add a new empty button row
  myKbd.addRow();
  // add a URL button to the second row of the inline keyboard
  myKbd.addButton("see docs", "https://github.com/shurillu/CTBot", CTBotKeyboardButtonURL);
}

void loop() {
  // a variable to store telegram message data
  TBMessage msg;

  // if there is an incoming message...
  if (myBot.getNewMessage(msg)) 
  {
    // check what kind of message I received
    if (msg.messageType == CTBotMessageText) 
    {
      // received a text message
      if (msg.text.equalsIgnoreCase("show keyboard")) 
      {
        // the user is asking to show the inline keyboard --> show it
        myBot.sendMessage(msg.sender.id, "Inline Keyboard", myKbd);
      }
      else 
      {
        // the user write anithing else --> show a hint message
        myBot.sendMessage(msg.sender.id, "Try 'show keyboard'");
      }
    } 
    else if (msg.messageType == CTBotMessageQuery)
    {
      // received a callback query message
      if (msg.callbackQueryData.equals(LIGHT_ON_CALLBACK))
      {
        // pushed "LIGHT ON" button...
        digitalWrite(led, LOW); // ...turn on the LED (inverted logic!)
        // terminate the callback with an alert message
        myBot.endQuery(msg.callbackQueryID, "Light on", true);
      } 
      else if (msg.callbackQueryData.equals(LIGHT_OFF_CALLBACK)) 
      {
        // pushed "LIGHT OFF" button...
        digitalWrite(led, HIGH); // ...turn off the LED (inverted logic!)
        // terminate the callback with a popup message
        myBot.endQuery(msg.callbackQueryID, "Light off");
      }
    }
  }
  // wait 500 milliseconds
  delay(500);
  Serial.println("Test");
} 

[hamster66]

Hi In my experience, you should not call this function more than once every 3 seconds. Chip 8266 is not very fast unfortunately.

[topsecretgtr]

Thank you for your reply. May I know how would you change the example above? The problem is that when getNewMessage is called, it lasted around 4-8 seconds, thus it also delayed other functions in the loop. Thus my temperature and humidity reading is refreshed every 4-8 seconds.

[shurillu]

Hello topsecretgtr,

Basically Telegram service is intended as istant messaging system for humans; if you want a realtime system to display data, this one may be not the good one. Anyway, as hamster66 says, the ESP8266 SOC is not very powerful (take in mind that all the communications are HTTPS connections -> encrypted). Today I'll release a new version that support the ESP32 SOC... maybe you can try with boards based on this chip.

Cheers,

Stefano

P.S.: you can speedup the execution (half second) by removing the delay function located in the end of the sketch: delay(500); <--- this!

[topsecretgtr]

@shurillu Thank you so much for your reply. Yes, I did remove the delay(500).

Actually, I used the Telegram service to monitor temp and humidity of a greenhouse remotely. I also have an LCD display to display the values realtime.

I don’t mind having a delay in obtaining temp and humidity through Telegram. But the delay also affect the readings on the LCD display. But, since its just for monitoring a greenhouse, I guess I could still live with the delay. Thank you for your great work.

[harry8914]

Same problem here.

[shurillu]

Hello Harry8914, as I said, actually it is impossible to speedup the communication process between Telegram server and the SOC (ESP8266/ESP32). If you need to update very quickly something (like reading sensors and updating a display) I suggest you to use another board (like an Arduino NANO) and use the ESP8266 only to send/receive the data. Another way is to use a dual core ESP32 board (pay attention: there are some single core SOC!): one core for time rilevant application and the other one for Telegram integration. Cheers

Stefano

[shurillu]

Hello, I've just published a new release (2.1.2) that mitigate the sluggishness of the getNewMessage() method execution (form around 4 seconds to one second). It's not a real asynchronous solution (I need to reconsider the whole library) but now with this trick it's four times faster.

Stefano

[midicraft]

Hello Stefano,

i noticed with the new Version (since 2.1.2) time for calling the getNewMessage() method is typically arround 1 second. Well done!

Sometimes though there is a case where the getNewMessage() is taking 15 seconds, the loop after 7 seconds the loop after 3 seconds and the loop after 1 sec. again. After those 4 loops it seems like the method is not executed for a few seconds(as wanted cause of the blocking feature) before this starts over again.

So worst case the program is stalling for about 25 sec. Which is ofcourse not ideal.

Any ideas on that?

I commented out the getNewMessage() part in my mainloop and that fixed it. So it's definitivly related to that method.

Best regards Alexander

[midicraft]

Ok here's the catch: When getNewMessage() takes longer than 350ms to complete it will be imediately triggered again in the next loop. Maybe it would be wise to reset the blocking timer at the end of this method to prevent this from happening.

That would ensure that there's a couple of loops executed after calling the method.

What do you think about that?

Also wold it be possible to just abort the method when it takes to long to get new messages from the server? That would ensure that the loop does not stall for more than a couple of seconds. I have seen runtimes for getNewMessage() up to 40 seconds. They are varying quite much and i don't know why yet...

Best reagrds Alexander

[shurillu]

Hello Alexander, very good point thank you! As you said, the timer must be cleared AFTER the Telegram response. Talking about your second question, I never encountered a long response delay by Telegram server so I never implemented something to "mitigate" these connection delays... thank you for reporting me this issue. So I added a timeout (should be in ms - documentation is very confusing) ad you can modify it by editing CTBotDefines.h - CTBOT_CONNECTION_TIMEOUT. Now is set to 2000 ms. If you want to try it please download the master branch and let me know, as I can draft a new release with these changes. Actually I can't simulate a connection timeout.

Cheers,

Stefano

[midicraft]

HI Stefano,

thanks for the quick fix! The Blocking now works as expected and allows for a couple of mainloops to happen before the next getNewMessage() is executed.

Also the added Timeout does it's job. I can now not observer loop times greater than a couple of seconds when the system is in idle. Ofcourse as soon as i am doing stuff like sending messages etc. the loop time becomes greater again. But that's expected.

I am happy for now. Thanks for your work it has really helped me push forward my current project. (Alarmbutton system for my grandpa) It handles all the messaging and some baisc menü functions like a charm.

Cheers Alex

[midicraft]

Hi Stefano,

here's an update from my side. Unfortunately the getNeMessage() is triggering a wdt reset sometimes now. I have narrowed the wdt reset down to the part where getNewMessage() is called.

So maybe wait with a new release until i have got some more infomartions.

EDIT: I commented out the new Timeout function again. For now htis seems to solve the issue.

WiFiClientSecureBearSSL.h (WiFiClientSecure) has timeout functionalitys build in. For SSL handshakes they allow up to 15 seconds. For server response up to 5 seconds.

I don't understand exactly why but somehow the added timeout functionality induces a state where wdt is triggered. Maybe you have an idea. If not i would suggest leave out the timeout for the next release as i am the only one for now having sporadic issues.

Maybe my issues are also related to my location. 2.4Ghz in a city crowded resedential area with about 100 wifis in reach is maybe a tough enviroment for an ESP8266

EDIT2: So i ran a test this night an wdt reset get's also triggered without the Timeout function. I'll dissable Software WDT for now when pulling messages. Worst case the hardware WDT is resetting the device after a couple of seconds.

Thanks again for the fix of the blocking! That improved things quite a bit :)

Best regards Alex

[shurillu]

Hello Alexander, maybe it can helps: I have just released a new version rebuilt from scratch: you can find it in the v3.0.0 branch. Basically it use the HTTPS POST in spite of the HTTPS GET: with this change, the library can keep the connection to the Telegram server active and it is not necessary bring it up for every call (like the old one does). I saw a big improvement in the time needed to execute a command (here where I live in Italy, around 200 ms, previously a couple of seconds - tipically 2-3 seconds). Another big improvement is that: no more String objects inside the class. The methods now use a standard C strings and there are overloaded version using String objects just for backward compatibility (internally they calls the methods with standard C strings). I have a sketch form a user that tipically do random resets: with this version seems to be stable. Only the data structures still use String objects. Finally I separated the Telegram calling methods (their names terminated with "Ex") from the method for reading Telegram server response. It allows to call a Telegram service (such as sendMessage) without decoding the Telegram server response: just call the method to drop the response after. There are still present the old methods for backward compatibility.

So if you wanna try this new version, just download it and recompile the sketch: it should compile and run without modification. In this case, let me know your experience ;-)

Cheers

Stefano

[midicraft]

Hi Stefano,

that sounds great! Unfortunately i have installed my creation on sunday and now have no easy acces to the hardware anymore. So far it runs pretty stable.

One question about the new server handling. If the controller is connected to the server at all times does that interfere with other libs that are requesting/sending from a server? I also use some simple IFTTT webhooks and NTPtimeclient alongside your bot library. I just imagine if the ESP is holding connecting that might block other traffic from happening. Or am i on the wrong track?

As soon as i am at my parents house for more than a day i'll play arround with v3 of the bot library and see if it can improve the stability even further.

Thanks for all of your hard work! Maybe consider ading a donate button somwhere :)

Best regards Alex

[100Gio]

@DaniBot