search

OpenZWave / Zwave2Mqtt

Fully configurable Zwave to MQTT gateway and Control Panel using NodeJS and Vue
MIT License
352 stars 93 forks source link

[bug] Error, ERROR: Not enough space in stream buffer #141

Closed ksuclipse closed 4 years ago

ksuclipse commented 4 years ago

Version

Build/Run method

Zwave2Mqtt version: 1.2.3 Openzwave Version: 1.4.1

Describe the bug after docker is running for a while (at least a day) i start to get many errors of Error, ERROR: Not enough space in stream buffer. This is solved by restarting the container. when this happens nothing talks on the zwave network.

To Reproduce Steps to reproduce the behavior: run docker for more than a day

Expected behavior Not this

Additional context Running commit: 85b49dcf00c3efdb3825e42d93b265052a9f58a6

robertsLando commented 4 years ago

Could you append log output?

Daniel

On 13 Oct 2019, at 03:44, Errol S. notifications@github.com wrote:

 Version

Build/Run method

[ X] Docker PKG [ X] Manually built (git clone - npm install - npm run build ) - manually build docker image Zwave2Mqtt version: 1.2.3 Openzwave Version: 1.4.1

Describe the bug after docker is running for a while (at least a day) i start to get many errors of Error, ERROR: Not enough space in stream buffer. This is solved by restarting the container. when this happens nothing talks on the zwave network.

To Reproduce Steps to reproduce the behavior: run docker for more than a day

Expected behavior Not this

Additional context Running commit: 85b49dc

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

Fishwaldo commented 4 years ago

This is usually seen on rPI or SBC boards with a Z-Stick. Using a USB powered hub usually resolves it.

(It’s probably related to the recently discovered incompatibilities discovered on rPI 4 boards. )

robertsLando commented 4 years ago

@ksuclipse Can I close this? Did you resolve your issue?

ksuclipse commented 4 years ago

yes not sure what happened but after recent pull its gone

ksuclipse commented 4 years ago

Happened again unfortunately. any thoughts?

OZW_Log.txt

ksuclipse commented 4 years ago

@robertsLando

robertsLando commented 4 years ago

@Fishwaldo could you check this?

ksuclipse commented 4 years ago

any ideas on this? seems to happen after several days of uptime. about to pull the newest commit so i'll keep you posted as well

Fishwaldo commented 4 years ago

See my comment above. It’s a hardware/kernel issue and not anything in OZW we can fix. Try a powered hub. Most reporters say they fix the issue.

ksuclipse commented 4 years ago

I’m directly connected to the nuc running it. Is there a better zwave stick I should be using??

johntdyer commented 4 years ago

I just started getting this as well on Home Assistant 0.102

Fishwaldo commented 4 years ago

As already mentioned, this is in most cases related to power issues with the USB port. Using a powered USB hub solves it most of the time.

ksuclipse commented 4 years ago

In my case at least it is directly connected to the board. Are you saying I should put a powered hub in between the pc and the zwave receiver??

On Wed, Dec 11, 2019 at 22:38 Justin Hammond notifications@github.com wrote:

As already mentioned this is in most cases related to power issues with the USB port. Using a powered USB hub solves it most of the time.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/OpenZWave/Zwave2Mqtt/issues/141?email_source=notifications&email_token=ABANZTEKFN2BIRSMHMWYIHTQYHE53A5CNFSM4JAFBWL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGVQXUQ#issuecomment-564857810, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABANZTGTZNHKGGG4HELVFITQYHE53ANCNFSM4JAFBWLQ .

Fishwaldo commented 4 years ago

Yes.

robertsLando commented 4 years ago

Just curious, is this problem also present when using aeotec z-stick usb gen5?

Daniel

On 12 Dec 2019, at 07:15, Justin Hammond notifications@github.com wrote:

 Yes.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Fishwaldo commented 4 years ago

yes - basically internally all sticks are the same... it affects most of them.

robertsLando commented 4 years ago

But why this happens with ozw 1.6 and not 1.4??

Daniel

On 12 Dec 2019, at 08:53, Justin Hammond notifications@github.com wrote:

 yes - basically internally all sticks are the same... it affects most of them.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Fishwaldo commented 4 years ago

Its happened since 1.0

It’s a hardware issue…

From: Daniel Lando notifications@github.com Sent: Thursday, 12 December 2019 4:48 PM To: OpenZWave/Zwave2Mqtt Zwave2Mqtt@noreply.github.com Cc: Justin Hammond justin@dynam.ac; Comment comment@noreply.github.com Subject: Re: [OpenZWave/Zwave2Mqtt] [bug] Error, ERROR: Not enough space in stream buffer (#141)

But why this happens with ozw 1.6 and not 1.4??

Daniel

On 12 Dec 2019, at 08:53, Justin Hammond <notifications@github.com mailto:notifications@github.com > wrote:

 yes - basically internally all sticks are the same... it affects most of them.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/OpenZWave/Zwave2Mqtt/issues/141?email_source=notifications&email_token=AAGNURBEMPVGQ56UVJARK5LQYH3CXA5CNFSM4JAFBWL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGV5HFQ#issuecomment-564908950 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGNURCQ2YOSUHL6UHNVZMDQYH3CXANCNFSM4JAFBWLQ . https://github.com/notifications/beacon/AAGNUREF2IDXCWXPNP6J5A3QYH3CXA5CNFSM4JAFBWL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGV5HFQ.gif

nellering commented 4 years ago

I just wanted to mention that I am getting this with an aeotec z-stick usb gen5 directly pluged into a Gigabyte 630M motherboard. I will also try a different port and pickup a powered usb hub to see if it fixes the issue.

johntdyer commented 4 years ago

@Fishwaldo

It’s a hardware issue…

Its a hardware issue as in our zWave sticks are broken ? Is there an upstream issue somewhere we can subscribe too for updates ?

nellering commented 4 years ago

I picked up an external powered USB hub, and I am still getting the buffer issue with my Aoetec Gen 5 plugged into the powered hub.

riemers commented 4 years ago

While reading up on things, also issues i can also recall having seen this before a few times. Have not seen it anymore after i started playing more with it. But i have a Nuc with mains power to it. If i see the same stream logs again i'll paste it here too.

riemers commented 4 years ago 
OpenZWave Error, ERROR: Not enough space in stream buffer
OpenZWave Error, ERROR: Not enough space in stream buffer
OpenZWave Error, ERROR: Not enough space in stream buffer
OpenZWave Error, ERROR: Not enough space in stream buffer

Was trying to do a fresh install again. Docker logs showed this beforehand. This was without any current zwcfg file. 

2019-12-28 16:16:35.605 Detail, Node089, Notification: NodeQueriesComplete
2019-12-28T15:16:35.610Z z2m:Zwave node 88 ready: Unknown: id=0154 - Unknown: type=0100, id=0201 (Unknown)
2019-12-28 16:16:35.612 Detail, Node090, Query Stage Complete (Dynamic)
2019-12-28 16:16:35.612 Detail, Node090, AdvanceQueries queryPending=0 queryRetries=0 queryStage=Configuration live=1
2019-12-28 16:16:35.612 Detail, Node090, QueryStage_Configuration
2019-12-28 16:16:35.612 Detail, Node090, QueryStage_Complete
2019-12-28 16:16:35.612 Warning, CheckCompletedNodeQueries m_allNodesQueried=0 m_awakeNodesQueried=0
2019-12-28 16:16:35.612 Warning, CheckCompletedNodeQueries all=0, deadFound=1 sleepingOnly=0
2019-12-28 16:16:35.612 Detail, Node090, Notification: NodeQueriesComplete
2019-12-28 16:16:35.612 Detail, Node092, Query Stage Complete (Dynamic)
2019-12-28 16:16:35.612 Detail, Node092, AdvanceQueries queryPending=0 queryRetries=0 queryStage=Configuration live=1
2019-12-28 16:16:35.612 Detail, Node092, QueryStage_Configuration
2019-12-28 16:16:35.612 Detail, Node092, QueryStage_Complete
2019-12-28 16:16:35.612 Warning, CheckCompletedNodeQueries m_allNodesQueried=0 m_awakeNodesQueried=0
2019-12-28 16:16:35.612 Warning, CheckCompletedNodeQueries all=0, deadFound=1 sleepingOnly=0
2019-12-28 16:16:35.612 Info, Failed to load device_classes.xml
2019-12-28 16:16:35.612 Info, Check that the config path provided when creating the Manager points to the correct location.
2019-12-28 16:16:35.612 Detail, Node092, Notification: NodeQueriesComplete
2019-12-28T15:16:35.614Z z2m:Zwave node 89 ready: Unknown: id=0154 - Unknown: type=0100, id=0201 (Unknown)
2019-12-28 16:16:35.614 Info, Failed to load device_classes.xml
2019-12-28 16:16:35.614 Info, Check that the config path provided when creating the Manager points to the correct location.
2019-12-28T15:16:35.615Z z2m:Zwave node 90 ready: Unknown: id=0154 - Unknown: type=0100, id=0201 (Unknown)
2019-12-28 16:16:35.615 Detail, Node093, Query Stage Complete (Dynamic)
2019-12-28 16:16:35.615 Detail, Node093, AdvanceQueries queryPending=0 queryRetries=0 queryStage=Configuration live=1
2019-12-28 16:16:35.615 Detail, Node093, QueryStage_Configuration
2019-12-28 16:16:35.615 Detail, Node093, QueryStage_Complete
2019-12-28 16:16:35.615 Warning, CheckCompletedNodeQueries m_allNodesQueried=0 m_awakeNodesQueried=0
2019-12-28 16:16:35.617 Info, Failed to load device_classes.xml
2019-12-28 16:16:35.617 Info, Check that the config path provided when creating the Manager points to the correct location.
2019-12-28T15:16:35.618Z z2m:Zwave node 92 ready: Unknown: id=0154 - Unknown: type=0100, id=0201 (Unknown)
--- multiple http calls ---
2019-12-28 16:24:01.039 Error, ERROR: Not enough space in stream buffer
2019-12-28 16:24:02.039 Error, ERROR: Not enough space in stream buffer
2019-12-28 16:24:02.039 Error, ERROR: Not enough space in stream buffer
2019-12-28 16:24:02.039 Error, ERROR: Not enough space in stream buffer
2019-12-28 16:24:02.039 Error, ERROR: Not enough space in stream buffer
2019-12-28 16:24:02.039 Error, ERROR: Not enough space in stream buffer
2019-12-28 16:24:02.039 Error, ERROR: Not enough space in stream buffer
etc etc etc
robertsLando commented 4 years ago

@Fishwaldo The problem seems still present after using a power hub

robertsLando commented 4 years ago

@Fishwaldo When upgrading to OZW 1.6 I'm getting this problem too, also I'm getting segmentation fault when sending switch multilevel set commands. This was working befor in OZW 1.4

Fishwaldo commented 4 years ago

The stream issue - it’s not a OZW issue. It’s the kernel running out of buffer space to transmit. It seems to be hardware based. I can’t do jack about that unless someone debugs the kernel/hardware.

The multilevel crash is known. The bug was present in OZW 1.4 but it didn’t crash. The fix is very involved and will be what I’m working on soon. (2 attempts at a fix caused other issues).

hyperbolic2346 commented 3 years ago

I'm hitting this as well and the WAF is painful. I originally hit this on a pi4 and found this issue. I tried a powered hub as well and no change. I then switched to a different machine, though low-power as well. Could this be related to the buffer defaulting to 2k and we fill that during high load?

https://github.com/OpenZWave/open-zwave/blob/f44387718c9f9509e410dc9c3e8be1aa2b0a5a74/cpp/src/platform/Controller.h#L67

I have no idea the message sizes we pass into this buffer or how many we can generate, but why not bump it and see if this disappears? Maybe add some metrics on the high water of the buffer?

robertsLando commented 3 years ago

@hyperbolic2346 are you using latest version of z2m? docker or manually built? what ozw version are you using?

hyperbolic2346 commented 3 years ago

I'm using the latest z2m docker image. It says 1.6.1240 in the control panel. Should I make a new bug since this one was closed?

If I knew how to build a new openzwave and map it into the docker container I would try this change out. An interesting note is that I put z2m on a different physical host that is much more powerful and haven't had the issue. This makes sense because the lower-power machines like the pi4 I was using earlier could get behind on the buffer, but I would expect it to be temporary and so the larger buffer could help there.

robertsLando commented 3 years ago

@hyperbolic2346 I suggest to open this issue on ozw repo, it's not a bug with z2m