search

WebThingsIO / gateway

WebThings Gateway - a self-hosted web application for monitoring and controlling a building over the web
http://webthings.io/gateway
Mozilla Public License 2.0
2.62k stars 339 forks source link

WebSocket sendMessage failed: Error: read ETIMEDOUT #3060

Open flatsiedatsie opened 1 year ago

flatsiedatsie commented 1 year ago

On the gateway 2.0 (Candle), I sometimes randomly get periods where the logs are empty. Mostly at night, for a few hours. This affects all addons.

missing_parts

After about two hours the system recovers.

How can I best debug what's causing this?

The logs indicate something happening almost every millisecond. Or is that just a result of how the log function works?

2023-01-04 02:30:00.465 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.466 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.467 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.468 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.469 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.470 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.470 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.471 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.472 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.473 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.474 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.475 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.475 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.476 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.477 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.478 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.479 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.479 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.480 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.481 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.482 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.483 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.484 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.484 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.485 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.486 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.487 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.488 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.489 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.489 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.493 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.494 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.495 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.495 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT
2023-01-04 02:30:00.496 ERROR  : WebSocket sendMessage failed: Error: read ETIMEDOUT

Once the system recovers the logs seem to indicate that the addons had no idea anything was wrong.

benfrancis commented 1 year ago

There isn't a 2.0 release yet, is this reproduceable on gateway's master branch?

It's a bit difficult to tell from this log alone but a wild guess would be that the gateway's IPC WebSocket connection with the add-on from which the property change messages are being sent is dropping for some reason. I'm not exactly sure how to debug that though.

flatsiedatsie commented 1 year ago

is this reproduceable on gateway's master branch?

Candle is pretty much the main branch, with only minor cosmetic changes. Reproducing it would probably entail making it a living system, as I suspect this happens when there are a certain amount of things and addons on the system.

a wild guess would be that the gateway's IPC WebSocket connection with the add-on from which the property change messages are being sent is dropping for some reason

Ah, so it could be that one addon is causing this? To be clear: once it happens incoming data from all addons is 'ignored'. The logs of things from various addons all show a hiatus during that period, not just the logs of things of one addon.

Could the use of execsync in addons perhaps cause this, forcing all the other addons to wait?