Increase Client timeout for ThingSpeak

[mrwee]

The current 1000ms client timeout is too short for e.g. communication over a cellular network (3G/4G). I propose the limit is increased to at least 5000ms (or better 10000ms) so slower / high latency connections also work.

[TD-er]

Hmm then we must also look into having enough calls to delay() or other measures to prevent the watchdog rebooting your device.

SW watchdog kicks in after roughly 2 seconds.

[mrwee]

Damn. I'm measuring roundtrip delays of ~3-4 seconds on a 4G router which doesn't work with ThingSpeak :( The 1 sec. is also pretty tight for some setups.

[ghtester]

Is it really an issue with timeout? Several years ago I was also struggling with sending data from more sensors to ThingSpeak and as far as I can remember, I had to send all sensors together by HTTP send command to get it working, like this: SendToHTTP api.thingspeak.com,80,/update?api_key=APIKEY=[Room1#Temperature]&field2=[Room2j#Temperature]
Otherwise ThingSpeak ignored a data from the 2nd sensor if it was sent too quickly after 1st one, asynchronously. The command above works reliably until today (ESP Easy Mega 2018-03-11) Hope this could help.

[mrwee]

I'm only sending one sensor data, and since it worked perfectly on my Wi-Fi/Fiber, but doesn't on the 4G router, then I can only assume that it's the latency / client timeout which causes this. Also from what I've read on the forum SendToHTTP isn't good for ThingSpeak due to the missing acknowledge. But thx for the inspiration :)

[TD-er]

Can you try to change the host name, save it and change it back? (while connected to the mobile, using the DNS from your provider) This will force a new DNS lookup and maybe it is using then a different IP.

Not sure if it will help, but it was the first thing that came to mind that could have an effect.

What other ThingSpeak protocols are there?

[mrwee]

I can try. I did both HW reset + reboot after it was switched to the 4G router. First Controller is OpenHAB MQTT (Working fine on 4G) Second is ThingSpeak.

But I just discovered that ThingSpeak actually supports MQTT. I think I'll try that out :)

[uzi18]

@mrwee what firmware file did you used? you got it in espeasy sysinfo subpage

[mrwee]

Apologize for the lack of this information. It's a pre-compiled version of mega-20200426 on an Wemos D1 mini Pro 2.0. Of course cleared with empty bin (16MB) first. Nothing else that the DS1820 and the two controllers configured. IP address: DHCP NTP configured

[uzi18]

normal 4M1M?

[mrwee]

Yes

[uzi18]

try this one (set more time) and give us connection statistics info: ESP_Easy_mega_20200512_normal_ESP8266_4M1M.zip

[mrwee]

ok, I will. I need to visit "the ESP" as it's located somewhere else. Would it be useful if I tested on Wi-Fi / Fiber with low latency?

[uzi18]

only if you want to compare later to 4g timeout

[mrwee]

ok. I've flashed a new D1 mini Pro 2.0 with this version and created a data push every 15 seconds to ThingSpeak, running on Wi-Fi/fiber. Nothing negative to report so far. Will try to go to the other device and re-flash as soon as possible.

[mrwee]

I have now tested with another 4G router since it's not easy to go on-site to the other. It seems like the new build is a bit unstable. It runs fine for some time, then the ESP becomes unresponsive and the serial log fills up with "1048249 : Error : Timeout while reading input data!"

This is regardless of client timeout set to 1000 or 4000. But maybe it's related to the interval setting for the dummy task publishing the data. I'll test further, as setting the interval to e.g. 30 mins, seems to make it crash somehow.

[TD-er]

Maybe the 4G connection is not via a NAT, but more like a proxy?

[mrwee]

Could be. My assumption was CGNAT. But yesterday I had huge ESPEasy stability issues on my three test devices, so I think I need to revert back to basics, and test again.

[TD-er]

CGNAT is really horrible. Try to get a stable VPN setup with CGNAT... So I guess that's a good candidate as source for these issues.

Also I've noticed quite a difference between providers, a few weeks ago. That was about maintaining a IPsec VPN. Via my mobile with a Vodafone SIM, it was next to impossible to create the VPN connection. With the same unit connected to another phone with a T-mobile SIM it worked flawless. Maybe the same filtering can also attribute to these issues you're seeing.

[mrwee]

Could be.. But the setup is quite basic. ESP/ESPEasy connected via Wi-Fi to Huawei 4G router. ESPEasy sends data to ThingSpeak (and an Mosquitto Broker for testing). But I also realize that I need to go on-site to troubleshoot. Not relevant trying to replicate with another router and another location.

MQTT from ESPEasy to Mosquitto does seem to be much more troublefree, than HTTP to ThingSpeak.

[ghtester]

If you see a stability issues when the MQTT is in use while there's a bad connectivity, it is a known issue (at least to me :-)) :

2684

So maybe you should better disable the MQTT and keep only ThingSpeak enabled which should be more stable.

[mrwee]

ok, that's good input. Reason for both was to validate that the data was picked-up and sent (somewhere), but if MQTT + ThingSpeak on instable connection, is a bad idea, then I'll remove MQTT. Hope ThingSpeak is stable then :)

[mrwee]

I have now tested the increased 2000ms client timeout for a week, and it works very stable. No missed measurements compared to the 1000ms setting. Thanks for the "quick fix". Edit: I've tested with only one controller (ThingSpeak). I don't know if that also makes a difference, compared to when I also had MQTT enabled.

[tonhuisman]

The issue seems to be resolved, so can be closed.