Message content filter

[jekyll2014]

Ciao Stefano, The problem I've found is toURL() method doesn't filter all characters to comply TELEGRAM requirements. Unfortunately I haven't found anly list of reserverd chars in TELEGRAM API but there are general recommendations on other resources. In my case I couldn't send help text message from my bot: "help\r\n" "get_sensor\r\n" "get_status\r\n" "get_config\r\n" "[ADMIN] set_time=yyyy.mm.dd hh:mm:ss\r\n" "[ADMIN] autoreport=0/1\r\n" "[ADMIN] set_output?=0/1\r\n" "[ADMIN][FLASH] set_interrupt_mode?=FALLING/RISING/CHANGE\r\n" "[ADMIN][FLASH] set_init_output?=on/off/x\r\n" "[ADMIN] getLog = uart/telnet/email/telegram\r\n" "[ADMIN][FLASH] check_period=n\r\n" "[ADMIN][FLASH] device_name=\r\n" "[ADMIN][FLASH] ssid=\r\n" "[ADMIN][FLASH] wifi_pass=\r\n" "[ADMIN] wifi_enable=1/0\r\n" "[ADMIN][FLASH] telnet_port=n\r\n" "[ADMIN][FLASH] telnet_enable=1/0\r\n" "[ADMIN][FLASH] http_port=n\r\n" "[ADMIN][FLASH] http_enable=1/0\r\n" "[ADMIN][FLASH] smtp_server=\r\n" "[ADMIN][FLASH] smtp_port=n\r\n" "[ADMIN][FLASH] smtp_login=\r\n" "[ADMIN][FLASH] smtp_pass=\r\n" "[ADMIN][FLASH] smtp_to=**@**.\r\n" "[ADMIN][FLASH] smtp_enable=1/0\r\n" "[ADMIN][FLASH] set_user?=n\r\n" "[ADMIN][FLASH] telegram_token=\r\n" "[ADMIN][FLASH] telegram_enable=1/0\r\n" "[ADMIN][FLASH] gscript_token=\r\n" "[ADMIN][FLASH] gscript_enable=1/0\r\n" "[ADMIN][FLASH] pushingbox_token=\r\n" "[ADMIN][FLASH] pushingbox_parameter=\r\n" "[ADMIN][FLASH] pushingbox_enable=1/0\r\n" "[ADMIN][FLASH] ntp_server=...\r\n" "[ADMIN][FLASH] ntp_time_zone=n\r\n" "[ADMIN][FLASH] ntp_refresh_delay=n\r\n" "[ADMIN][FLASH] ntp_enable=1/0\r\n" "[ADMIN][FLASH] set_event?=condition:action1;action2\r\n" "[ADMIN][FLASH] events_enable=1/0\r\n" "[ADMIN][FLASH] set_schedule?=time:action1;action2\r\n" "[ADMIN][FLASH] scheduler_enable=1/0\r\n" "[ADMIN][FLASH] display_refresh=n\r\n" "[ADMIN][FLASH] sleep_enable=1/0\r\n" "[ADMIN] reset\r\n" Until I implemented my own prefiltreing like: tmpStr.replace(F("\r"), ""); tmpStr.replace(F("\n"), F("%0a")); tmpStr.replace(F("\t"), F("%09")); tmpStr.replace(F(" "), F("%20")); tmpStr.replace(F("["), F("%5b")); tmpStr.replace(F("]"), F("%5d")); tmpStr.replace(F("?"), F("%3f")); tmpStr.replace(F("@"), F("%40")); tmpStr.replace(F("&"), F("%26")); tmpStr.replace(F("\"), F("5c")); tmpStr.replace(F("/"), F("%2f")); Not sure all of them are necessary but definitely "\r", "\n", "[", "]" are... Hope to investigate it later and make a list of "a must" chars to filter.

[shurillu]

Hello Andrey, it's a known issue: it's involving on the URL encoding. The problem is identify all the characters that must be encoded (cfr. https://www.w3schools.com/tags/ref_urlencode.asp). Actually, the library filter the "\n" character and the space " " character. All the others are left unchanged.

In these weeks I'll be very busy, but this is in the "to do list".

Stefano

[array81]

Any news about this?

[jekyll2014]

Ciao Stefano,

I'm sorry - been very busy with Moscow CrossCountry running RFID checking project... The competition will be run. Now most of the programming is done and I hope to do some tests. Will update ASAP.

[jekyll2014]

Ciao Stefano, It was my mistake to post a test code into another issue (#21)... I'll delete my message from issue #21.

The matter is some chars block the whole message from being sent. I've made a test which sends every ASCII byte one by one in a separate message to test which char is prohibited but ithe result is frustrating - all chars from 0 till 127 been sent and received on my mobile so I believe the problem is not the character but character combination and ASCII->UTF encoding.

P.S. Can't understand why chars from 128 till 255 haven't been sent... Will try to dig in you code deeper.

Here is the test sketch:

include "ESP8266WiFi.h"

include "CTBot.h"

CTBot myBot;

String ssid = ""; // REPLACE mySSID WITH YOUR WIFI SSID String pass = ""; // REPLACE myPassword YOUR WIFI PASSWORD, IF ANY String telegramToken = "***"; // REPLACE myToken WITH YOUR TELEGRAM BOT TOKEN TBMessage msg;

String command; bool uartReady = false;

uint32 senderID = **;

byte b = 0; bool sending = true; void setup() { // initialize the Serial Serial.begin(115200); Serial.println("Starting TelegramBot..."); // connect the ESP8266 to the desired access point myBot.wifiConnect(ssid, pass); // set the telegram bot token myBot.setTelegramToken(telegramToken); myBot.useDNS(true); }

void loop() { //check UART for data while (Serial.available()) { char c = Serial.read(); if (c == '\r' || c == '\n') uartReady = true; else command += (char)c; }

//process data from UART if (uartReady && command != "") { String str = processCommand(command); Serial.println(str); command = ""; uartReady = false; }

if (WiFi.status() == WL_CONNECTED && sending) { if (!myBot.testConnection()) Serial.println("testConnection ERROR"); else { //Serial.println("testConnection OK"); String str = "post="; str += "Char 0x"; str += String((int)b, HEX); str += " = <"; str += String((char)b) + ">"; String s = processCommand(str); Serial.println(s); if (b < 255) b++; else sending = false; delay(3000);

} } else { Serial.println("\nWiFi not connected"); delay(1000); } }

String processCommand(String command) { String str = "";

if (command.startsWith("senderID=")) { senderID = command.substring(command.indexOf('=') + 1).toInt(); str = "New senderID = "" + String(senderID) + """; } else if (command.startsWith("post=")) { String message = command.substring(command.indexOf('=') + 1); str = "Sending to [" + String(senderID) + "]: "" + message + """; sendToTelegram(senderID, message); } else if (command.startsWith("start=")) { String message = command.substring(command.indexOf('=') + 1); b = message.toInt(); str = "Start sending from byte : 0x" + String((int)b, HEX); sending = true; } else if (command == "stop") { String message = command.substring(command.indexOf('=') + 1); str = "Stop sending"; sending = false; } else { str = "Incorrect command: "" + command + """; } return str; }

void sendToTelegram(uint32 user, String str) { if (WiFi.status() == WL_CONNECTED) { if (!myBot.testConnection()) Serial.println("testConnection ERROR"); else { myBot.sendMessage(user, "Bot>> " + str); } } else { Serial.println("\nWiFi not connected"); } }

[shurillu]

Hello Andrey, as I answered in the other issue #21 (check it please), Telegram is not built for handle binary messages (it can handle binary data but the data must be encoded). If you want to send extended characters (not binary data!) as chinese characters (logograms) for example, use the UTF8 coding. If you still want to send binary data, you have to encode it in a way that Telegram can handle, for example Base64 (as I said in the issue #21) and obviously, you have to decode it once received. Actually, if a message contains ASCII codes that Telegram can't handle, the message will be rejected by the server. Keep in mind that the message you want to send is a part of a HTTP GET method so, speaking simple, it's like an URL and in your browser bar you only put text characters. In a future update, I'll do a check in the sendMessage() method and as it can return false if the message is not deliverable. Anyway the sendMessage() method is not intended as a "binary message sender": it send only text messages and "╚" (ALT + 200 -> 0xC8h) for example is not a text character. I hope to have clarified how thesendMessage() work.

Cheers,

Stefano

[jekyll2014]

Ciao Stefano,

A'm not talking about binary packages - I'm talking about the characters which stops message from being sent/delivered. You can try to send help message from my initial post and you'll see that it's not possible to send. So does some other messages containing non-numerical/alphabet characters like {}[]()~!@#$%^&*. I just wanted to understand if some of the characters are not allowed to send as plain text (only as %xx hex code) but it doesn't seems that easy now...

[shurillu]

Hello Andrey, I have just uploaded in the master branch some fixes that correct this issue: now you can (should) send any kind of characters. Forgive me for waiting, but I was a very busy with work. Let me know if the issue is solved as I can release a new version.

Stefano