Server application crashes regularly on Windows

[SiJiL82]

As title, the server app crashes regularly. Only way I notice is glancing at the dial and see it's not moving, then check in Task Manager and see the process is no longer there. Are there any logs anywhere to try and diagnose why it's crashing?

[SasaKaranovic]

Hi @SiJiL82

Thank you for reporting this. VU Server is logging to file.

If you are on Windows it should be located at c:\Users\USERNAME\vudials\server.log or if you are on Linux it should be /home/USERNAME/vudials.log

[SiJiL82]

Nothing in the logs when it happens. At some point in the past hour the server app has stopped running, but all that's in the logs is the startup info.

2024-01-28 09:21:46,370 INFO set_logger_level(61) Logging level: INFO
2024-01-28 09:21:46,415 INFO __init__(508) Loading server config...
2024-01-28 09:21:46,416 INFO __init__(19) VU1 config yaml file: C:\Program Files (x86)\KaranovicResearch\VUDials\_internal\config.yaml
2024-01-28 09:21:46,417 INFO __init__(17) VU1 Database file: C:\Program Files (x86)\KaranovicResearch\VUDials\_internal\vudials.db
2024-01-28 09:21:47,241 INFO __init__(522) VU1 HUB port: COM3 - USB Serial Port (COM3)
2024-01-28 09:21:49,049 INFO __init__(26) Communication timeout set to 10 seconds
2024-01-28 09:21:49,234 INFO __init__(528) No additional dials found. Searching the bus for new ones...
2024-01-28 09:21:50,102 INFO run_forever(564) Karanovic Research Dials - Starting API server
2024-01-28 09:21:50,103 INFO run_forever(572) VU1 API server is listening on http://localhost:5340
2024-01-28 09:21:50,119 INFO run_forever(576) Master Key is present in config.yaml
2024-01-28 09:21:50,120 INFO run_forever(577) Provide 'API_KEY' to your main application.
2024-01-28 09:21:50,120 INFO run_forever(578) to allow it to manage this server and the VU dials.

[SiJiL82]

I see in the code it supports "debug" as a logging level - how do I start the server up with this mode?

[SasaKaranovic]

You can open the command prompt and navigate to the VU Server installation directory (where VUServer.exe is located). Then run VUServer.exe --logging debug and it should output more verbose logging.

Note that if you disconnect dials or your USB gets reset/re-enumerated it will cause the server to crash.

[SiJiL82]

Nothing interesting in the logs I don't think? vudials.log

[SasaKaranovic]

The log seems to be incomplete. Have you attached the entire log file? But nothing of interest in the portion you attached.

[SiJiL82]

No, the log has been split up into multiple parts, and I figured only the most recent part would be useful. Here's the other parts - had to rename them as GitHub doesn't allow you to attach files with .1 or .2 as the file extension vudials-1.log vudials-2.log

[SiJiL82]

Joined the log files together into 1 file to make it easier. Don't know why I didn't think of this before, sorry!

vudials-complete.log

[SasaKaranovic]

PR #22 should hopefully make this issue easier to catch and debug.

[SasaKaranovic]

New version of the server app should log as well as show UI dialog to inform user that application has encountered a problem. Closing this issue and we can reopen if it becomes relevant again.

[SiJiL82]

I'm not getting any logs at all with the new version. It also still crashes just as regularly, with no UI dialog.

[SiJiL82]

I just removed and reinstalled the server app. When I started it up, I got a "VU Server has crashed unexpectedly!" popup, but it didn't crash - it started up and is still running fine. Still getting no logs generated though.

[SasaKaranovic]

@SiJiL82 Are you running the VU Server app as the administrator?

[SiJiL82]

No, it's running as my user account. I've also run the code manually, rather than run the Windows compiled install, and that does the same thing and doesn't save any logs

[SiJiL82]

Ah. The path has changed! You mentioned c:\Users\USERNAME\vudials\server.log in an earlier reply, it's now C:\Users\USERNAME\KaranovicResearch\vudials!

[SasaKaranovic]

Yes, the logging path has been changed in recent PR. Do you get any entries in the log file?

No, it's running as my user account.

VU Server app should be run with administrative privileges. Otherwise you might get issues.

[SiJiL82]

It is running with administrative privileges - just not as the administrator account. I've launched it 2 different ways to test - either just running the shortcut created by the installer, which pops up a UAC prompt to elevate it, or via Task Scheduler, with "Run with highest privileges" checked. Both ways still crash, with no popup generated and no log entries that detail why it crashed - so I assume it's crashing at a point that needs some exception handling that doesn't currently exist to make it throw the cause somewhere?

[SasaKaranovic]

I have created a new branch that should give you a pop-up dialog with the error that we catch.

Can you please try installing VU Server from the installer that is made on this separate branch -> https://github.com/SasaKaranovic/VU-Server/actions/runs/7910338075

My assumption is that either there is a permission issue with the folder or com port access problem. But let's see what error message you get and hopefully that tells us what is the issue.

[SiJiL82]

Still nothing with that branch. No popup first, nothing in the logs - it just stopped.

[SasaKaranovic]

@SiJiL82 Are you using the installer and running the .exe file or running from branch source code?

[SiJiL82]

I used the installer yesterday, it crashed after about 1h30m with no errors in the logs and no popup message. Today I've been running the source code directly, and it has been running for 11 hours without crashing... So maybe something with how it's compiled into an exe that's the problem?

[SasaKaranovic]

When you get a chance, can you attach your full log file.

The app should behave the same way if it's compiled or run from source code. Can you try connecting the dials to a different USB? If you are using a hub, can you try instead connecting them directly to the PC.

[SiJiL82]

joinedFile.txt It's missing the start because with debug on it fills the 2 files quite quickly.

I do have the dial connected via a hub, I'll get a longer USB-C cable and try it directly into the PC

[SasaKaranovic]

I don't think you need to run with the debug flag on. I suspect the issue is on the USB bus. If your dials/hub get disconnected or your computer goes into suspend/sleep mode, the VU Server app might crash since dials will get disconnected. There is an open issue to get this fixed. Also connecting directly to the PC is advised if feasible (but definitely not a requirement).

[SiJiL82]

Been running it for a couple of weeks with the dial connected directly to my PC, still get the same problem - the server app crashes with no logs or popups. It runs for less than an hour each time. Running the python code directly has no issues at all - it's run for 11 hours straight without crashing. I've stopped running the compiled app now and just launching the python file as a scheduled task instead.

[SasaKaranovic]

Thanks @SiJiL82 for the information and sorry about the inconvenience. I will re-open this issue since running application from source is not really a work-around for compiled version crashing. :)

I have created a new branch and here are build artifacts that you can try to use -> https://github.com/SasaKaranovic/VU-Server/actions/runs/8217219628

This version is identical to what is currently in master but binary is compiled with out --no-console option, so whenever you run VU Server you should see a window with all the printouts. Very annoying for production but for debugging it should be helpful.

Please download the VU1_Server artifact from this link and extract it to a folder. Close the VU Server if it's already running. Then open a command prompt, navigate to the folder you extracted the .zip contents to and type VUServer.exe and press enter. Now you should see all of the VU Server output and hopefully the last message before the server crashes. Please let me know what is the output before the app crashes.

p.s. Please run the app via command prompt or power shell. If you run it directly, you will get the console output but as soon as the application finishes/crashes the console window will close and you won't be able to copy the output.

[SiJiL82]

So I've been running that server version for a while now, and it's not crashed once. Runs all day with no issues. Swapped back to the normal version a few times, swapped USB ports, swapped cables - always crashes after a max couple of hours with no error messages and nothing in the logs. Going back to the version with the console shown again - no issues.

Very odd!

[SasaKaranovic]

Thanks for testing it out!

That's interesting result. The only difference between the two is if that one has the main window hidden and the other does not. Is there any chance that you have some AV/optimization turned off that would kill the VU Server app? Other than that I'm not really sure what could be causing this.

[github-actions[bot]]

This issue is stale because it has been open 15 days with no activity. Remove stale label or comment or this will be closed in 3 days.