[BUG] Companion crashing with ERR_STREAM_DESTROYED error

phillipivan commented 3 weeks ago

Is this a bug in companion itself or a module?

[X] I believe this to be a bug in companion

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug

We have had companion crash twice this afternoon reporting the same error both times. Once while it was sitting 'idle' without operator interaction.

Steps To Reproduce

Unclear how to reproduce at the moment.

Expected Behavior

Not crashing.

Environment (please complete the following information)

- OS: Windows 10 Pro 22H2
- Browser: Chrome - 124.0.6367.79
- Companion Version: 3.3 Stable

Additional context

Running with 27 active modules and 9 disabled modules.

Julusian commented 3 weeks ago

Any idea on what could have been going on when this happened? Unfortunately this error is incredibly vague (it almost just says 'an error occured'), so it might require a lot of trial and error to figure out where this is coming from

phillipivan commented 3 weeks ago

At the moment, no. The companion logs I looked at afterwards werent especially revealing. There were some logged errors from modules, but nothing scary, and some entries about a streamdeck being disconnected (via satellite). I am not especially convinced any of these are contributing factors. I suppose the other potentially noteworthy factors:

Its running on a VM inside a Windows Server instance
It is a system we are testing with a centralized companion instance and satellite for stream decks around the facility. So its not in production yet, and fingers crossed we haven't seen this issue in our production systems.
It has quite a few more modules than any of our production systems, and some of those modules are in dev, so probably have their own bugs.
The error message was identical both times it crashed today, with the same code line references.

As a test and dev system for now we can keep running and monitoring to see how it goes. But does Companion write a stack trace anywhere on crash that we can access and share? Or can we switch on a higher level of internal logging?

2024-06-04T01:03:58.872Z error Surface/Handler/CL21L2A02850 Satellite StreamDeck: xl disconnected 2024-06-04T01:03:58.873Z info Service/Satellite/::ffff:172.20.145.69:49921 connection closed with 1 connected surfaces.

Logs, such as they are, attached.

Companion Logs.zip

krocheck commented 3 weeks ago

This would seemingly have to be during an attempted file write operation, possibly not having write permissions to a directory? Since this is a dev VM, I'd try launching Companion as an admin and see if that resolves it. Maybe also (before doing that) compare the files and folders in the config directory to those on a Win desktop and see if there's a noticeable difference or something missing on the dev server.

Julusian commented 3 weeks ago

With some help from @krocheck I think we have figured out where this is coming from. So this is fixed in the betas and will be in 3.3.1 when that is done

phillipivan commented 3 weeks ago

Which build is it in? We will spin it up on the VM and see if the problem reoccurs.

krocheck commented 3 weeks ago

3.4.0+7035-main-5f79fdbc or later in the betas. v3.3.1 is in the pipeline for release by the end of the week.

phillipivan commented 3 weeks ago

@krocheck is there still value in testing Companion run at Admin level today? Happy to do that if it will be of any use.

krocheck commented 3 weeks ago

@phillipivan, @Julusian was able to reproduce in a test case once we found where this could come from (an outside module for log rotation). I'd prefer to see if it goes away with a patched version than muddy the waters, since running as admin could null the file system level condition that's causing normal operation to throw the error. At minimum, there should be a Companion log for the error condition rather than an exception thrown.

phillipivan commented 3 weeks ago

No worries, we will run up the beta build and confirm it is solved.

Kudos both for the speedy attention.

phillipivan commented 3 weeks ago

We have not encountered this issue today running either the beta or 3.3.1.

bitfocus / companion