Default scan_intervals overwhelm the W610, causes problems in home assistant

[canton7]

From work done on https://github.com/nathanmarlor/foxess_modbus, we've found what the W610 supports one read about every 700ms (sometimes it can do one quicker). It's capable of reading 8 registers at a time, but the modbus integration only reads 1 register at a time. This means that it theoretically takes 28s to read the 40 sensors in modbusUSB.yaml.

modbusUSB.yaml contains a range of scan_intervals, but they add up to more than 1 read every 700ms.

This means that the modbus integration spawns read operations more quickly than they can be completed. There's no logic to say "I want to refresh this sensor, but it's still waiting for the previous refresh to finish, so I'll hold off" -- it just keeps throwing read operations onto a queue inside home assistant. The queue gradually grows in size, which is going to slow things down, and might eventually crash home assistant. It also delays things like shutdowns and restarts. It also means that sensors aren't actually read at the interval that was requested.

However there's no indication of this unless you go and manually add logging calls to the modbus integration. I did this, and very quickly I've got to having tasks which have been sitting in the queue for 15 minutes+, and it's still rising.

To make matters worse, some people are discussing setting the scan_interval to 5 seconds, or even 1 second, without knowing the problems this causes.

You should probably increase the scan_intervals in modbusUSB.yaml so that they don't mess up home assistant for people who are using a W610 by default, and add some strong warnings not to decrease them?

[TonyM1958]

Good work, see #120 and the scan intervals suggested in there. This comes from direct experience that the default scan intervals also crash the modbus server connected to the inverter ethernet port, so this sounds similar to the problem with W610. If these devices can only process 1 scan every 700ms, it might also help explain why delay:1 also stops the modbus server from crashing as it forces a 1 second wait. This won't stop a queue building up in HA, although I haven't, yet, seen a problem with HA, just the inverter returning connection refused.

[canton7]

I commented on there as well. A scan interval of 5s is far too fast for the W610 when you're using the modbus integration; even 10s only lets you have maybe 13 sensors, rather than the 40 in modbusUSB.yaml.

The delay makes no difference. That just ignores refreshes for the first N seconds after the connection is established, so it postpones the issue for all of 1 second.

[TonyM1958]

5s is what they are currently set to - that was the status quo, but that is not what I am advocating.

Just counted up: in my modbusLAN.yaml, I have 7 sensors running with a 10 second scan_interval, 10 sensors running at 1 minute and 6 running at 5 minutes - so 23 in total.

There are more sensors available over RS485. In modbusUSB.yaml, I have 8 sensors running at 10 second scan_interval, 16 sensors running at 1 minute and 30 running at 5 minutes (so, well over 40 after you add the 11069 counters).

Its not correct to say that delay makes no difference if you are using the inverter LAN port directly - this definitely stops ithe Fox modbus server from crashing. It might not make a difference for anyone using W610.

The real issue here, for me, is how to update the released code scan_intervals - that is what #120 was raised for. For example, CT2 Meter in modbusUSB.yaml has a scan interval of 1 second and battery cell voltages are scanned very 5 seconds.

[canton7]

I have 8 sensors running at 10 second scan_interval, 16 sensors running at 1 minute and 30 running at 5 minute

Right, I think that's almost enough to overwhelm the W610. You'd have to hack in some logging to see for sure.

Its not correct to say that delay makes no difference if you are using the inverter LAN port directly

This issue is about overwhelming the W610, and my comment there was in relation to overwhelming the W610. I am not talking about the direct LAN connection at all, because that is not what this issue is about.

[TonyM1958]

I have 8 sensors running at 10 second scan_interval, 16 sensors running at 1 minute and 30 running at 5 minute

Right, I think that's almost enough to overwhelm the W610. You'd have to hack in some logging to see for sure.

If W610 can process 1 read every 700ms, you can process over 400 reads in 5 minutes.

• 8 sensors every 10 seconds = 240 reads in 5 minutes
• 16 sensors every 60 seconds = 80 reads in 5 minutes
• 30 sensors reads in 5 minutes
• total of 350 sensor reads in 5 minutes

So, any queue will empty in less than 5 minutes and there won't be a build up.

The point I was making was to generically group the scan intervals and parameterise them to make it easier to adjust. Hence, the suggested division into high, medium and low scan rates. Once you have done this, users can adjust their own intervals without changing the core yaml files, making updating easier and allowing people using W610 to reduce the scan rates or others to increase them as they wish.

[canton7]

My point, and the reason I opened this issue, is:

1. The default scan intervals overwhelm the W610
2. There's almost no indication of when the user has set their scan intervals too low

Splitting the sensors into low/medium/high is fine, but if users then go "Well, I want to see my PV1 more often, I'll just decrease the "fast" interval to 1s", they'll shoot themselves in the foot without ever realising it.

(All of this is moot if you have an integration which doesn't just keep blindly firing off read tasks. You can also send more than one read at a time to the W610 which improves this massively, and you can talk to it over UDP rather than TCP which also speeds it up. Over at foxess_modbus we're able to read all registers every in about 2 seconds even with the W610, which means there's no need to faff around with categorising sensors into fast/medium/slow. But, I'm being nice and warning users of this integration that the default setup is overwhelming HA.)

[TonyM1958]

Good work, see #120 and the scan intervals suggested in there

Which is why I suggested a way to change the scan intervals that could resolve this (not sure why we are so violently agreeing with each other here??)

[canton7]

They're separate, but related, issues. You could fix this issue and ignore #120, and you could implement #120 without consideration for this issue (or both!). For example, you could quickly fix modbusUSB.yaml now, while the discussion around ideal scan times in #120 was still ongoing.

I'm trying to stop this issue becoming a duplicate discussion on adding fast/medium/slow scan intervals - #120 is the home for that discussion. I'd like to keep this issue focused on highlighting the fact that people are being told to buy the W610 and then given a configuration which isn't suitable for that adapter. If this turns into a parallel discussion to #120, that message gets lost.

[StealthChesnut]

@canton7 thanks for this - it was a concern I originally had with the inverter itself (and ballooning HA db sizes!). Now that the number of sensors has grown so much I will lengthen the intervals.

I was going to implement multiple sensor reads, so I'm glad you mention the W610 can only do groups of 8. Useful to know. As Nathan's integration already does a lot of the next steps I was going to do, I will likely point to his as a preferable option at least for now.

[canton7]

Thanks! It can normally do reads of 10, but these fail a small amount of the time. We found 8 was the largest read size it could do reliably. It looks like this is a bug in the firmware rather than a deliberate restriction, as it sends back a packet which is malformed.

Compare that to almost every other adapter, which can happily do reads of 50-100!

[StealthChesnut]

I was working direct with the inverter, or through the ethernet port (until it was disabled, and I haven't been able to get the firmware updated to the latest yet to fix it), so I would never have known about it!

I'll look at updating this integration to do multiple reads as I have got that working, but it does take a bit of time - I have to pause my own HA install to do development work, and the other "users" get annoyed when the house "doesn't work"!

On Mon, 17 Apr 2023 at 14:00, Antony Male @.***> wrote:

Thanks! It can normally do reads of 10, but these fail a small amount of the time. We found 8 was the largest read size it could do reliably. It looks like this is a bug in the firmware rather than a deliberate restriction, as it sends pack a packet which is malformed.

Compare that to almost every other adapter, which can happily do reads of 50-100!

— Reply to this email directly, view it on GitHub https://github.com/StealthChesnut/HA-FoxESS-Modbus/issues/123#issuecomment-1511295637, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALFO3FG6VWXAUH5BGLREORLXBU5INANCNFSM6AAAAAAXABYPOI . You are receiving this because you commented.Message ID: @.***>

[screenagerbe]

I have to pause my own HA install to do development work, and the other "users" get annoyed when the house "doesn't work"!

That's the main reason why I created a clone of my HA container. You only have to disable the specific integration/device/script in PROD while you're working on it in TST. It's a bit of double work, but all my integrations are tested in TST for about a week or 2 and then I'll port them to PROD.

[canton7]

I do development on foxess_modbus in a dev container. Easy to spin one up with an installation of HA, with your config inside.