search

arendst / Tasmota

Alternative firmware for ESP8266 and ESP32 based devices with easy configuration using webUI, OTA updates, automation using timers or rules, expandability and entirely local control over MQTT, HTTP, Serial or KNX. Full documentation at
https://tasmota.github.io/docs
GNU General Public License v3.0
22.19k stars 4.81k forks source link

websend always delays receiving a http reply to itself for 5 seconds #14306

Closed pilaGit closed 2 years ago

pilaGit commented 2 years ago

PROBLEM DESCRIPTION

A clear and concise description of what the problem is.

After a websend executes, that particular device will not receive/handle a http reply to that same device immediately, but will always add 5 sec delay. Only then will it falsely report {"WebSend":"Connect failed"} and continue with normal and correct handling of a http reply.

REQUESTED INFORMATION

Make sure your have performed every step and checked the applicable boxes before submitting your issue. Thank you!

- [ ] If using rules, provide the output of this command: `Backlog Rule1; Rule2; Rule3`:
```lua
  Rules output here:

- [ ] Set `weblog` to 4 and then, when you experience your issue, provide the output of the Console log:
```lua
  Console output here:

TO REPRODUCE

Steps to reproduce the behavior:

from the console at 2.60 directly back to the same 2.60 - pointless, but easiest to demonstrate the 5 seconds lag

websend [192.168.2.60] Power 1

17:42:20.548 CMD: websend [192.168.2.60] Power 1 17:42:25.695 RSL: RESULT = {"WebSend":"Connect failed"}

from 2.60 To the web server 2.10 with http reply returning back to the same 2.60

websend [192.168.2.10] /?x=z60

17:05:37.443 CMD: websend [192.168.2.10] /?x=z60
17:05:42.784 RSL: RESULT = {"WebSend":"Connect failed"}

EXPECTED BEHAVIOUR

A clear and concise description of what you expected to happen.

After issuing a websend, command must be correctly confirmed as done and a http reply must be executed immediately (with no 5 seconds delay added). It should work like here, when the command does not return to the same device 2.60 but to the 2.71

from 2.60 Zigbee Bridge directly to the 2.71

websend [192.168.2.71] Power 1

12:42:11.609 CMD: websend [192.168.2.71] Power 1
12:42:12.300 RSL: RESULT = {"WebSend":"Done"}

from 2.60 Zigbee Bridge to the web server sending http reply to 2.71

websend [192.168.2.10] /?x=z71

17:09:01.923 CMD: websend [192.168.2.10] /?x=z71
17:09:03.677 RSL: RESULT = {"WebSend":"Done"}

SCREENSHOTS

If applicable, add screenshots to help explain your problem.

ADDITIONAL CONTEXT

Add any other context about the problem here.

Problem was noticed at a Zigbee bridge when testing sending Zigbee button clicks to my SmartHome program to execute appropriate commands by sending http reply to the Tasmota devices.

When a resulting reply command had to be sent back to that same Zigbeee bridge, it would always result in 5 seconds wait on the bridge and a false "Connect failed". Then it would be executed normally, as that did not occur.

If the same websend leads to http reply being sent to any other device, all is perfectly well and quick (1.5-2 seconds).

In my tests, Zigbee keypresses are sent to a Webserver which just forwards them to my SmartHome program. SmartHome program then sends needed http commands back to the Tasmota devices directly.

Same behavior on: several very different devices I tested with current and older Tasmota. Tested: Sonof ZBbridge, Shelly Bulb Duo, Athom POW.

(Please, remember to close the issue when the problem has been addressed)

pilaGit commented 2 years ago

Just to prove that everything works when websend is not followed by a http reply to the same device. Sending the same On command from 2.71 to the Zigbee bulb at 2.60

from the console at 2.71 to the 2.60 Zigbee bridge

WebSend [192.168.2.60] ZbSend { "Device":"IkeaSijalica", "Send":{"Power":"on"} } 

13:01:02.142 CMD: WebSend [192.168.2.60] ZbSend { "Device":"IkeaSijalica", "Send":{"Power":"on"} }
13:01:03.778 RSL: RESULT = {"WebSend":"Done"}

and at 2.60 we get:

13:01:03.298 RSL: RESULT = {"ZbSend":"Done"}
13:01:04.025 RSL: SENSOR = {"IkeaSijalica":{"Device":"0xF3D0","Name":"IkeaSijalica","Power":1,"Endpoint":1,"LinkQuality":60}}

from 2.71 to the web server 2.10 with http reply returning to the 2.60. From issuing a command at 2.71 to the bulb reaction at 2.60: 1.819 sec

websend [192.168.2.10] /?x=z60

13:10:51.507 CMD: websend [192.168.2.10] /?x=z60
13:10:52.860 RSL: RESULT = {"WebSend":"Done"}

at 2.60 we see:

13:10:53.326 RSL: RESULT = {"ZbSend":"Done"}
13:10:54.039 RSL: SENSOR = {"IkeaSijalica":{"Device":"0xF3D0","Name":"IkeaSijalica","Power":1,"Endpoint":1,"LinkQuality":60}}
sfromis commented 2 years ago

Unclear what you really mean by always delays receiving a http reply and then go on to provide examples of having no such delay.

Are you expecting a confirmation/error while the other server is still processing? WebSend has a timeout of 5 seconds if waiting for a response, as tested by calling a server set to have a longer response time. As Tasmota cannot know if the server has an intention of responding or not, it is not possible to immediately detect the bad server behavior of just not responding. Arguably, 5 second is a bit quick for a timeout, but that's a separate question.

Your test case with a bad request to the Tasmota device itself merely exhibits that Tasmota handling of this senseless scenario is not great, nothing more.

pilaGit commented 2 years ago

As the title says. When a Tasmota Zigbee bridge 2.60 sends out webrequest and the http reply is immediately returned back to that same 2.60:

a) delay of 5 seconds is inevitable - problem one b) web send is falsely claimed as failed - problem two

No 5 sec delay is expected behavior which occurs when websend is not followed by http replay to the same device.

sfromis commented 2 years ago

Nothing about Zigbee in the title.

Is the short version that if you, from a Zigbee bridge, sends a request to a server providing a quick well-formed response, that you will get a delay of 5 seconds, and then a failure.

Or is it that you have a badly behaving server which does not bother to respond to requests, and you just want Tasmota to not detect if the WebSend was successful or not?

As Tasmota WebSend has the feature of telling you if the send was done or failed, that logically includes waiting for a response.

pilaGit commented 2 years ago

Zigbee is where I encountered the problem. Problem exists in all devices. Zigbee bridge and Zigbee itself is not the problem at all.

This first thing thing you write is the problem. Tasmota does insert delay.

As for my programming, you can see in addendum that it took 1,8 seconds to execute everything correctly when one device (2.71) is sending a web request to my program that will be terminated on another device (2.60) which is the Zigbee bridge.

I have seen people complaining about 5 seconds delay in websend. I have never seen someone defining a clear test case.

sfromis commented 2 years ago

You are using the words "when websend is not followed by http replay to the same device". This is not my first point, where I specifically asked about a scenario with a "server providing a quick well-formed response".

When using a ZbBridge to do a WebSend to a "server providing a quick well-formed response", I saw no delay. When I made the server response time longer than 5 seconds, Tasmota reported failure after 5 seconds. This makes sense, as long as you accept such a brief timeout.

pilaGit commented 2 years ago

My server response time is always the same, under 0,5 sec for this issue. My additional example shows that the entire cycle is performed in 1,8 seconds when websend goes from one Tasmota and reply to another.

When that perfectly same chain is started with websend from the one Tasmota and generates http reply to that same Tasmota, the 5 sec delay is introduced.

sfromis commented 2 years ago

Again: A Tasmota device doing a WebSend to itself is not a valid test case, as it is not well-behaved. Or was this not what you meant?

Basically, I'm still trying to make you define a clear test case, which is not a Tasmota device doing a WebSend to itself, but a well-behaved server (possibly another Tasmota device). In my testing, a WebSend to another Tasmota device generates a quick "Done", just as expected.

pilaGit commented 2 years ago

I wrote that directly sending to itself was pointless! Just shows 5 seconds lag.

The problem is: the device sends websend to another device and then from another device this same device (the first one recives http reply.

Use Zigbee as an example. It has a Zigbee button and a Zigbee bulb. Zigbee button sends command pressed. Tasmota forwards that to the main external program. That external program reads that input, and returns the http command: turn on the Zigbee light at that same Zigbee Bridge.

pilaGit commented 2 years ago

OK, here is the complete test setup that shows a problem. Everything needs to be executable, adjust folders as needed

To your web server add: zbtest.php

<?php
$inputArgs = $_GET['x'] ;
$output = shell_exec("bash /root/ztest.sh $inputArgs 2>&1 ");
echo "$output";
?>

Then, create a bash script ztest.sh

#!/bin/bash

case "$1" in
(z60) curl -s http://192.168.2.60/cm?cmnd=ZbSend%20%7B%20%22Device%22%3A%22IkeaSijalica%22%2c%20%22Send%22%3A%7B%22Power%22%3A%22on%22%7D%20%7D
    # SendHttp "60" 'ZbSend { "Device":"IkeaSijalica", "Send":{"Power":"on"} }'
;;
(z71)
    curl -s http://192.168.2.71/cm?cmnd=Power%201
    # SendHttp "71" "Power 1"
;;
esac

Web server is 2.10. At 2.60 there is a Zigbee bridge with a Zigbee bulb (IkeaSijalica) on it. At 2.71 there is a Tasmota bulb.

Connect to the Zigbee bridge (ANY Tasmota device will do as long as the reply goes back to it). Issue the following two commands in the console of 2.60.

websend [192.168.2.10] /zbtest.php?x=z71 websend [192.168.2.10] /zbtest.php?x=z60

I get the result:

14:58:40.514 CMD: websend [192.168.2.10] /zbtest.php?x=z71
14:58:41.496 RSL: RESULT = {"WebSend":"Done"}
14:58:48.906 CMD: websend [192.168.2.10] /zbtest.php?x=z60
14:58:53.920 RSL: RESULT = {"WebSend":"Connect failed"}
14:58:54.275 RSL: RESULT = {"ZbSend":"Done"}
14:58:54.973 RSL: SENSOR = {"IkeaSijalica":{"Device":"0xF3D0","Name":"IkeaSijalica","Power":1,"Endpoint":1,"LinkQuality":63}}

When http reply goes back to the oroginal device, a 5 sec delay is added and {"WebSend":"Connect failed"} is false alert.

pilaGit commented 2 years ago

If I issue the same two commands, but this time from the console of 2.71, the same thing, but now the problem is at 2.71. Zigbee is irrelevant here.

websend [192.168.2.10] /zbtest.php?x=z71 websend [192.168.2.10] /zbtest.php?x=z60

15:21:25.615 CMD: websend [192.168.2.10] /zbtest.php?x=z71
15:21:30.926 RSL: RESULT = {"WebSend":"Connect failed"}
15:21:31.241 RSL: RESULT = {"POWER":"ON"}
15:21:31.244 RSL: POWER = ON
15:21:43.141 CMD: websend [192.168.2.10] /zbtest.php?x=z60
15:21:43.854 RSL: RESULT = {"WebSend":"Done"}
s-hadinger commented 2 years ago

Got it. This is a race condition. WebSend is a blocking command. It stops everything in Tasmota until the request returns a response or the timeout happens.

In your example, the WebSend blocks and prevents from responding to 2.71. When the timeout occurs, 2.71 keeps retrying and eventually the request goes through. But the connection from 2.60 is already broken so no answer gets back.

There is no plan to move WebSend to async. Without this, this kind of usage is not possible.

I would strongly suggest to move to MQTT which is async by nature.

sfromis commented 2 years ago

While you indeed wrote "pointless", you still used that example it to make the point of a 5 second delay, and also used the word "reply" a lot without meaning a reply to a http request.

Seems that the misunderstanding is that when you used the word "reply" (and "http reply") you did not mean a reply to the http request, but instead the unusual behavior of creating a new independent http request from the server, in addition to the server doing the regular protocol reply to the original http request. Without Tasmota being multi-threading, problems are not surprising.

Anyway, the http part of Tasmota is not meant as a robust server part, and is not well-tested or tuned. It indeed seems that Tasmota receiving a http request while WebSend is still waiting for a response can make it fail.

There are reasons why MQTT is in the name of Tasmota, as this is a much better communication protocol instead of the mess of assuming WebSend to be a good API. MQTT is what Tasmota is designed for.

pilaGit commented 2 years ago

Just pick any two Tasmota devices, and send them Power 1 or any other command.

#!/bin/bash

curl -s http://192.168.2.$1/cm?cmnd=Power%201
    # SendHttp "$1" "Power 1"

Issue commands:

websend [192.168.2.10] /zbtest.php?x=71 websend [192.168.2.10] /zbtest.php?x=60

When you websend and receive to and from the same device: delay and false reporting.

websend [192.168.2.10] /zbtest.php?x=71 websend [192.168.2.10] /zbtest.php?x=72

from 2.71

15:37:44.766 CMD: websend [192.168.2.10] /zbtest.php?x=71
15:37:49.783 RSL: RESULT = {"WebSend":"Connect failed"}
15:37:50.021 RSL: RESULT = {"POWER":"ON"}
15:37:50.023 RSL: POWER = ON
15:37:54.158 CMD: websend [192.168.2.10] /zbtest.php?x=72
15:37:54.866 RSL: RESULT = {"WebSend":"Done"}
pilaGit commented 2 years ago

OK, I understand this will not be fixed.

As for MQTT being better than http: I have a very complex SmartHome systems running perfectly using only bidirectional http communication. In multiple locations, running everything for several years now. Just one single SmartHome program and mostly Tasmotas. I bought bulbs just now to test Zigbee. I needed Zibee to add switches.

There are ways around this, as in most cases.

I guess I should close this issue.

sfromis commented 2 years ago

Not using MQTT, you are losing out on a number of Tasmota features, in addition to http generally being slower, and with more risk of not being quite reliable.

pilaGit commented 2 years ago

Speed is not a problem. reliability is perfect, my program has no bugs. Nothing to maintain. I never found a feature I needed I could not use via http. My SmartHome system is very complex.

The only problem with the Tasmota that I have, is not related to this but inability to corrcetly re/set power meters. Everything else I fixed in my program.

That being said, Tasmota works amazingly good! Perfectly stable, no problems with it once you identify probablems and fix them. Tasmota is truly amazing piece of software!

And yes, Tasmota works perfectly with just a poor http :) I would only never use websend to send critical data directly from one to another Tasmota device - it may not be received.

The problem I reported here was found in my testing of Zigbee. In reality, it can be easily avoided.

pilaGit commented 2 years ago

Speed is not a problem. reliability is perfect, my program has no bugs. Nothing to maintain. I never found a feature I needed I could not use via http. My SmartHome systems are very complex.

The only problem with the Tasmota that I have, is not related to this but inability to corrcetly re/set power meters. Everything else I fixed in my program.

That being said, Tasmota works amazingly good! Perfectly stable, no problems with it once you identify problems and fix them. Tasmota is truly amazing piece of software!

And yes, Tasmota works perfectly with just a poor http :) I would only never use websend to send critical data directly from one to another Tasmota device only once - it may not be received. Handshake or a repeat could solve this if needed.

The problem I reported here was found in my testing of Zigbee. In reality, it can be easily avoided.

pilaGit commented 2 years ago

OK, now that I understand that problem. I would like to propose an alternative solution. Quite simple to implement.

Allow user to define timeout for websend on the fly. At least let us turn it off completely.

websend0 should need one or two lines of code to modify timeout from 5 seconds to none. Problem solved.

Since there is already inbuilt support to parse websend0, websend7 and similar, this could be used to define the timeout precisely.

That way, the user becomes responsible for the timeout if so desired. Websend can be better utilized. For all other users, the command remains the same: websend with 5 seconds timeout. Not a problem.

Feature quickly added. One exotic problem solved.

ascillato2 commented 2 years ago

Moving to Ideas as this is now a feature request. Thanks.