Multiple Topics changed via HTTP-Requests: Some skipped.

[Riemen]

If more than one topic is changed in Heishamon simultaneously, some are not processed correctly.

I created a button in Loxone to force hot water generation manually, even at night when operation mode is at 0 or 1 (Heat only/Cool only). To do this I send the following commands via HTTP-requests:

/command?SetDHWTemp=55 /command?SetOperationMode=5 One second later (to make sure this is after operation mode change) /command?SetForceDHW=1 (This is only one example. The same seem to happen with other commands.)

It seems like all commands are recieved by Heishamon correctly. But only the first command is processed. The others seem to written to a buffer and later be skipped due to a timeout?

Here is the corresponding log entry:

Fri Aug 30 22:53:17 2024 (69861): set DHW temperature to 55 Fri Aug 30 22:53:17 2024 (69862): sent bytes: 111 including checksum value: 219 Fri Aug 30 22:53:17 2024 (69942): set heat pump mode to 35 Fri Aug 30 22:53:17 2024 (69943): Already sending data. Buffering this send request Fri Aug 30 22:53:17 2024 (70312): Received 203 bytes data Fri Aug 30 22:53:17 2024 (70313): Checksum and header received ok! Fri Aug 30 22:53:17 2024 (70314): received TOP1 Pump_Flow: 30.21 Fri Aug 30 22:53:17 2024 (70318): received TOP53 Eva_Outlet_Temp: 34 Fri Aug 30 22:53:17 2024 (70323): Sending command from buffer Fri Aug 30 22:53:17 2024 (70324): sent bytes: 111 including checksum value: 111 Fri Aug 30 22:53:18 2024 (70851): set force DHW mode to 128 Fri Aug 30 22:53:18 2024 (70851): Already sending data. Buffering this send request Fri Aug 30 22:53:19 2024 (72327): Previous read data attempt failed due to timeout! Fri Aug 30 22:53:19 2024 (72327): Received 0 bytes data Fri Aug 30 22:53:19 2024 (72328): Sending command from buffer Fri Aug 30 22:53:19 2024 (72329): sent bytes: 111 including checksum value: 18

IMHO entries in buffer should never be skipped, even if a timeout occurs. Also the timeout seems to be way to low. how can it be increased?

I am not sure if the commands collide with reading of the heat pump data. I already increased the reading interval to 30 seconds, but the issue remains. It still occurs every time, even if the requests arrive between the read intervals.

[IgorYbema]

The timeout has nothing to do with the buffering. The command was sent fine:

Fri Aug 30 22:53:19 2024 (72328): Sending command from buffer

However it is good practisch to check from your automation if a set command has been changing the parameter you wanted to change and if not keeps sending it after say 10 secs. Commands can get lost due to multiple reasons

[IgorYbema]

Let me break down what the logs mean, changing the order a bit:

First the part to set the DWH temp. That was fine. The command was sent and the resulting answer was received (after a command the heatpump always sends a status response back, not always having the changed data in there by the way). The only changed topics are the pump flow and the eva outlet temp. Fri Aug 30 22:53:17 2024 (69861): set DHW temperature to 55 Fri Aug 30 22:53:17 2024 (69862): sent bytes: 111 including checksum value: 219 Fri Aug 30 22:53:17 2024 (70312): Received 203 bytes data Fri Aug 30 22:53:17 2024 (70313): Checksum and header received ok! Fri Aug 30 22:53:17 2024 (70314): received TOP1 Pump_Flow: 30.21 Fri Aug 30 22:53:17 2024 (70318): received TOP53 Eva_Outlet_Temp: 34

Then, while sending and receiving the previous part, the heatpump mode set was received. It got buffered because of the previous part didn't end yet. As soon as the previous part was finished the command was sent (from out of the buffer). However, it didn't receive any response. It is not known if the command was successfully received by the heatpump or not, but heishamon didn't get the regular status response (after waiting 2 secs). This can happend and will resulst in a lower good data percentage. Fri Aug 30 22:53:17 2024 (69942): set heat pump mode to 35 Fri Aug 30 22:53:17 2024 (69943): Already sending data. Buffering this send request Fri Aug 30 22:53:17 2024 (70323): Sending command from buffer Fri Aug 30 22:53:17 2024 (70324): sent bytes: 111 including checksum value: 111 Fri Aug 30 22:53:19 2024 (72327): Previous read data attempt failed due to timeout! Fri Aug 30 22:53:19 2024 (72327): Received 0 bytes data

While waiting for that 2nd send/receive, the third command was received and therefore buffered also. Just after the 2 sec timeout of the not-received status response this command was sent also. The log ends here so can't tell if the status response from this command was received correctly. But as I said the sending isn't coupled to the receive response as most often the changed parameter is only visible in the next query. If the log was a bit longer we should have seen at least the DHW temp change and probably also the 2nd command, the mode set. The 3rd command change would not yet get acknowled as it would have been seend in a later response. Fri Aug 30 22:53:18 2024 (70851): set force DHW mode to 128 Fri Aug 30 22:53:18 2024 (70851): Already sending data. Buffering this send request Fri Aug 30 22:53:19 2024 (72328): Sending command from buffer Fri Aug 30 22:53:19 2024 (72329): sent bytes: 111 including checksum value: 18

This brings be back to my previous message. Don't asume that when sending a command to heishamon it will always result in a successfull change of the paramater. Your automation must check if the parameter got changed and if not, it then must send the command again until it get changed.

My own automation does this by having a table of parameters internally which are changed by the automation logic so the logic can alway continu. Then there is another routine which checks these internal parameters and match them with the status from the heatpump/heishamon. If the parameters are off, they get changed with a set command. This way the logic and the commands are uncoupled and I am sure that the paramaters will eventually get changed.

[McMagellan]

See here: https://github.com/Egyras/HeishaMon/issues/410

[Riemen]

Hello Igor,

I get your reasoning.

But is it really the best solution to leave it to the users to build logic to check if their commands are set properly? Depending on which system they use, doing so can be very complicated. Wouldn't it be better, if HeishaMon would do this by itself? I mean you already have all needed data available and programming this would not be very complicated:

Any received commands are written to and kept in buffer, even after they have been sent to the heat pump. (It would suffice to keep the latest command for any topic. For Example if 2 commands SetForceDHW=1 and SetForceDHW=0 are received, the second command would replace the first in buffer.)

After the next receiving of values from heat pump, these are synchronized with the values in buffer. If both match, the values are removed from buffer. If not, the command is being resent to the heat pump. This is done, until no more commands are in buffer.

This way, you could make sure that received commands are always transferred to the heat pump correctly.

(Based on the thin data of above example, a command seems to be processed reliably by the heat pump, if HeishaMon receives a response? I am not sure, if this is always the case or if there are instances where a response is received although the command is not set correctly? If the first is the case, the above logic would only be needed if no response is received after passing of a command.)

Best regards Tobias

[IgorYbema]

It could but that is beyond the original design which we agreed upon in the early stages. And in any case the MQTT or web commmand can be lost also so the automation should check also. So better to keep it there.

[Riemen]

HeishaMon sends responses with 200 OK Codes for all command requests. You would notice if any HTTP request is lost. So this is no valid objection. Although I don't know how this is handled via MQTT.

Also my proposed change would not invalidate any existing solutions. It would only improve the reliability of internal command handling. And by reducing the lost commands basically to zero, all future API calls to HeishaMon could be clearly simplified.

Maybe you (and your team) could at least reconsider the proposed improvement for a future release.

[IgorYbema]

We could do that but we agreed uppon that heishamon is a gateway and doesn't store any state information (execpt for the rules engine). Also the previous model is written to be safe on the memory side. The command buffer only accepts 5 commands in total in the buffer for example for that reason and it is just not smart to do all that checking also. The newer model can have more buffer due to more memory though. But rewrite the code so it always checks if a command results in the expected behaviour isn't on the goals for the near future. For now just keep in mind to have that logic on the user side.

Also bare in mind that indeed the 200 OK get received but then, if you don't get a 200 OK the automation software you use still needs to handle this anyway (for example if heishamon lost its wifi for a short time). So a good practice to recheck the resulting behaviour from the automation side is needed anyway. This also has another effect. If someone modifies the heatpump parameters outside of your automation (like on the controller or the panasonic website), it will get changed back immediatly after your automation recevies that new status as it doesn't match with the internal state which it thinks it should be in.

[mi-hol]

we agreed uppon that heishamon is a gateway and doesn't store any state information (execpt for the rules engine)

@IgorYbema I'm new to HeishaMon and knowing about this design decision and the resulting consequences for usage is a very import topic! Unfortunately its not mentioned in readme or at least not found via a GH search in other files. Would you mind to add this and maybe other decisions to the readme?

[HighlyCompressedAir]

I think it's clear now. Heishamon doesn't check if command succeded or not, it's up to the automation to make sure of it. This could be added to readme as it's confusing. Other than that this issue can be closed.

[geduxas]

I think it's clear now. Heishamon doesn't check if command succeded or not, it's up to the automation to make sure of it. This could be added to readme as it's confusing. Other than that this issue can be closed.

I think it should be clear enough, that heishamon is just only interface which translates one language to another.. it's just bridge.. communication device.. yes it's has rules, but those are very new

[HighlyCompressedAir]

What about optional pcb commands though like SetSmartGridMode or SetBufferTemp for example? Can they fail? My understanding is that these commands are sent to the heat pump continously, so there's no need for checks, but I'm not sure.