Linux player development should be abandoned in favor of making the Windows player more compatible with Wine

Esinko commented 1 year ago

Is your feature request related to a problem? Please describe. The Linux player is severely outdated and cannot display modern web content.

Describe the solution you'd like The development of the Linux player should be abandoned in favor of making the Windows player more Wine compatible. At this time, I've been able to run the Windows player (version v3-R301.1) with Wine (7.17-staging) using the dotnet472 and dxvk libraries installed with Winetricks. There are only a few issues currently:

The watchdog program does not work at all. Either it gets stuck in a loop of spawning processes or starts leaking memory, causing the Linux kernel to terminate Wine all together once available memory runs out.
Stability issues with random crashes. I'm still looking for the causes of some weird instability issues. I will update my findings and solutions later to this issue if anyone needs them.

The advantages of abandoning the Linux player in favor of making the Windows player more compatible with Wine should be self-explanatory: the development team can focus on only the Windows player with minimal overhead, while supporting Windows and Linux platforms.

Describe alternatives you've considered In theory just continuing work on the Linux player would be the ultimate solution, however, there does not seem to be any interest to do so.

Additional context

dotnet472 is dependent on bunch of other .Net versions. Installing is a pain. Lots of clicking continue, but just one command with Winetricks.
The Windows player does not run on Wine with Wine Mono. Fixing this would make running the Windows player on Wine much more user-friendly. Link: https://wiki.winehq.org/Mono
dxvk is a compatibility library to convert DirectX calls to Vulkan calls. This is required for the player to be stable at runtime.
Wine 7.17-staging must be used, because the stable version of Wine does not support the networking implementation of the Windows player at this time.
Winetricks is a utility script for Wine: https://wiki.winehq.org/Winetricks

Short tutorial: How to run the Xibo Windows player with Wine

Note: Desktop of some sort required. I used plain X11.

Install Wine staging version 7-17 (or later!) https://wiki.winehq.org/Wine_User%27s_Guide#Getting_Wine
Install Winetricks https://github.com/Winetricks/winetricks/releases
Install cabextract (https://www.cabextract.org.uk/) and GStreamer (32-bit, https://gstreamer.freedesktop.org/documentation/installing/on-linux.html?gi-language=c)
Install dotnet472 and corefonts winetricks -q dotnet472 corefonts
Install dxvk winetricks -q dxvk
Make sure Wine is configured as Windows 10 winecfg -v win10
Download the Xibo Windows player installer and run it wget "https://github.com/xibosignage/xibo-dotnetclient/releases/download/3R301/xibo-client-v3-R301.1-win32-x86.msi" wine ./xibo-client-v3-R301.1-win32-x86.msi 6.1. Disable the Watchdog by renaming the executable To know why, read the comments above mv "$HOME/.wine/drive_c/Program Files (x86)/Xibo Player/watchdog/x64/XiboClientWatchdog.exe" "$HOME/.wine/drive_c/Program Files (x86)/Xibo Player/watchdog/x64/DXiboClientWatchdog.exe" mv "$HOME/.wine/drive_c/Program Files (x86)/Xibo Player/watchdog/x86/XiboClientWatchdog.exe" "$HOME/.wine/drive_c/Program Files (x86)/Xibo Player/watchdog/x86/DXiboClientWatchdog.exe"
Start the player wine "$HOME/.wine/drive_c/Program Files (x86)/Xibo Player/XiboClient.exe" append o after the command as an argument to start the options program

dasgarner commented 1 year ago

You are correct that we are struggling to make any improvements to the Linux player, and we need to make a decision regarding it's future.

Running under WINE is an interesting idea and certainly something we've explored in the past. Last time we tried we encountered a few issues, some of which you've already identified:

stability
the browser engine we use (cefsharp) didn't work at all the last time we tried
problems with video codecs

I'd be very interested to know if the browser functionality works in your testing?

Thanks!

Esinko commented 1 year ago

Browser functionality does indeed work. I haven't had any issues with it after installing dxvk. The issue is Wine's DirectX support, using Vulkan fixes that.

Is there any video codecs with known issues I could test?

dasgarner commented 1 year ago

Ah, that's interesting! It gets us much closer to a functional player then, even if there are some stability issues (which we do also have on the existing Linux Player).

Regarding video, we had problems with anything that wasn't a run of the mill H.264 MP4. We haven't tested recently though, so it might be that this is resolved. In any case we can recommend a codec.

The Windows player does not run on Wine with Wine Mono. Fixing this would make running the Windows player on Wine much more user-friendly.

I think this will be difficult as we're heavily invested in WPF in terms of how the player layers its content and renders into frames, etc.

Wine 7.17-staging must be used, because the stable version of Wine does not support the networking implementation of the Windows player at this time.

I wonder if we could take a modular approach to this and convert the networking functionality (XMDS, XMR, web server, etc) into a .NET core application which can has native support on Windows/Linux. The WPF section (which we run under WINE) handles the rendering, but doesn't require any networking capability.

Esinko commented 1 year ago

I wonder if we could take a modular approach to this and convert the networking functionality (XMDS, XMR, web server, etc) into a .NET core application which can has native support on Windows/Linux. The WPF section (which we run under WINE) handles the rendering, but doesn't require any networking capability.

I don't specifically remember what the issue with Wine's stable version was anymore. However, I think development efforts should be focused on something else as Wine will eventually support the current networking implementation in its stable version. If you want to dig-in on the actual issue, simply try to start the player with the current stable version of Wine and otherwise follow my guide above.

I'll do some testing with different video codecs and I'll post any further developments in terms of stability improvements.

Esinko commented 1 year ago

I should clarify that my goal in this issue is to move focus to making the Windows player run better on Wine, but also easier to install.

dasgarner commented 1 year ago

Understood thank you for your efforts.

I'll set aside some time to test on Ubuntu 22.04 under Wayland and see if any other issues pop up as a result.

Esinko commented 1 year ago

I've done some more testing and I've found the sole culprit of the stability issues at the moment.

Currently the player seems to stay stable for about 2 hours, after which XiboClient.exe tries to access or write to an invalid memory address:

Code: c0000005
Description: Unhandled page fault on write access to 00000004 at 7BC61E1E
Thread: 013c

"A C0000005 error is memory error. Specifically, a C0000005 error is an access violation error caused by a buffer overrun", says Microsoft.

There are some errors related to SMB networking. Does Xibo use that directly? Otherwise, this might be a defiance in Wine or a bug in the Xibo Windows player. In any case, my next step is to fix the SMB networking issues anyway, which will probably involve installing a distro-specific winbind package.

dasgarner commented 1 year ago

There are some errors related to SMB networking. Does Xibo use that directly?

No, Xibo does not use SMB directly.

We have seen various memory related bugs in the past, although they always need to originate in the layer below our code. For example we saw the old browser engine throw all sorts of memory issues out.

Esinko commented 1 year ago

Interesting. I'll see if I can find some more details on under what conditions these crashes happen and if they are related to any certain content being displayed.

EDIT: The issue is being raised at runtime and might be related to the change of layout. However, it seems Winetricks has improperly configured which .NET versions Wine should use. I'll have to redo most of my testing and see if some further trickery is required during the installation process to fix these misconfigurations. However, my current theory is that I accidentally broke some configs while re-installing various dependencies.

Esinko commented 1 year ago

At this time, the player is reporting 100% uptime (with only browser content) in the last day after I reinstalled .NET (see edit in last post) and the missing SMB networking dependencies. I'll move on to testing video content.

EDIT: The Xibo dashboard was showing incorrect results. The player crashed 4 times in 24h.

Esinko commented 1 year ago

There are some errors related to SMB networking. Does Xibo use that directly?

No, Xibo does not use SMB directly.

We have seen various memory related bugs in the past, although they always need to originate in the layer below our code. For example we saw the old browser engine throw all sorts of memory issues out.

I have more details on this crash (I'll post a full stack soon). CrBrowserMain.exe under XiboClient.exe causes the c0000005 crash. I suspect this is the embedded browser that Xibo uses?

Esinko commented 1 year ago

@dasgarner I have an update on video playback support.

So far I've been able to get past issues related to missing encoders and such. However, I've hit a dead end. My PC I use for testing has an iGPU that is too old (Intel Haswell generation). In short, I need someone else to test video playback.

I'm going to update my original issue with some things I've changed to get as far as I am now.

dasgarner commented 1 year ago

EDIT: The Xibo dashboard was showing incorrect results. The player crashed 4 times in 24h.

The up time report is based on whether the player checks in at the configured frequency, so its certainly possible it crashed and restarted before it missed its window.

I have more details on this crash (I'll post a full stack soon). CrBrowserMain.exe under XiboClient.exe causes the c0000005 crash. I suspect this is the embedded browser that Xibo uses?

This is the underlying Chromium process from CEF. The stack goes Xibo -> CefSharp -> CEF. Changing layout, or changing widget in a region will tear down and recreate a browser.

In short, I need someone else to test video playback. I'm going to update my original issue with some things I've changed to get as far as I am now.

I've got a Linux machine set up in our lab which I can use to do some video testing. I won't be there until Wednesday unfortunately, but once I've set it up I can get access more easily.

dasgarner commented 1 year ago

Well I have no idea what I am doing 😄 following the instructions (all worked OK) up until running Xibo gives me the attached output.

wine_terminal_log.txt on Ubuntu 20.04.

Wine staging is now 7.22 and I couldn't work out how to install the older one. Xibo is on v3 R307.

In any case I have TeamViewer on the machine so I can poke around with it over the holiday period.

Esinko commented 1 year ago

Well I have no idea what I am doing 😄 following the instructions (all worked OK) up until running Xibo gives me the attached output.

wine_terminal_log.txt on Ubuntu 20.04.

Wine staging is now 7.22 and I couldn't work out how to install the older one. Xibo is on v3 R307.

In any case I have TeamViewer on the machine so I can poke around with it over the holiday period.

Wine staging should work fine as well. I went through my notes and the new component in the staging branch we need is winsock support, which seems to have persisted.

Esinko commented 1 year ago

Well I have no idea what I am doing 😄 following the instructions (all worked OK) up until running Xibo gives me the attached output.

wine_terminal_log.txt on Ubuntu 20.04.

Wine staging is now 7.22 and I couldn't work out how to install the older one. Xibo is on v3 R307.

In any case I have TeamViewer on the machine so I can poke around with it over the holiday period.

I've seen this issue before. Try to do what is said here.

dasgarner commented 1 year ago

I've seen this issue before. Try to do what is said https://github.com/doitsujin/dxvk/issues/2384#issuecomment-1111290717.

Thanks, that got it running! 👍

I'll do some testing and report back.

dasgarner commented 1 year ago

OK so for web content (text, tickers, etc) the CEF sub process crashes with this message:

OnRenderProcessTerminate: a cef sub process has terminated. ProcessCrashed

The display management functions all seem to be working well - all the files downloaded and XMR is connected.

Images display fine.

Videos don't display and go to the fault state with this message, which means VFW_E_CANNOT_RENDER. The video is a MP4 which does play natively on the OS.

Video Failed: Exception from HRESULT: 0x80040218

Esinko commented 1 year ago

OK so for web content (text, tickers, etc) the CEF sub process crashes with this message:

OnRenderProcessTerminate: a cef sub process has terminated. ProcessCrashed

The display management functions all seem to be working well - all the files downloaded and XMR is connected.

Images display fine.

Videos don't display and go to the fault state with this message, which means VFW_E_CANNOT_RENDER. The video is a MP4 which does play natively on the OS.

Video Failed: Exception from HRESULT: 0x80040218

Could I get the full log related to that cef crash? That video error is new to me. I'll have to look into it.

EDIT: Actually, could you provide full (individual) logs of trying to display video and trying to display browser content?

dasgarner commented 1 year ago

EDIT: Actually, could you provide full (individual) logs of trying to display video and trying to display browser content?

I'm trying to get some more detail here, but it doesn't seem like anything else is logged within the player itself.

Here are the terminal logs. wine_terminal_log.txt. I couldn't separate the two events out unfortunately - but if we look at the video one first, the other one should then be isolated.

There are messages in there from Gstreamer about a missing quicktime codec - although the video is a MP4:

It looks like the CefSharp.BrowserSubprocess.exe process is getting the same OpenVR error I had previously with the main application.

Esinko commented 1 year ago

EDIT: Actually, could you provide full (individual) logs of trying to display video and trying to display browser content?

I'm trying to get some more detail here, but it doesn't seem like anything else is logged within the player itself.

Here are the terminal logs. wine_terminal_log.txt. I couldn't separate the two events out unfortunately - but if we look at the video one first, the other one should then be isolated.

There are messages in there from Gstreamer about a missing quicktime codec - although the video is a MP4:

It looks like the CefSharp.BrowserSubprocess.exe process is getting the same OpenVR error I had previously with the main application.

Everything related to OpenVR should just be disabled. I'll try to figure out how to do that. However, it seems your Quardo P400 is unsupported by Vulkan... or has outdated drivers:

warn: Skipping Vulkan 1.1 adapter: llvmpipe (LLVM 12.0.0, 256 bits)
warn: Skipping Vulkan 1.2 adapter: Quadro P400
warn: DXVK: No adapters found. Please check your device filter settings and Vulkan setup.
info: Process set as DPI aware
ERR: DisplayD3D.cpp:158 (CreateRendererD3D): Failed to create D3D renderer: Failed to get device caps, HRESULT: 0x887 6086C ERR: Display.cpp:1004 (initialize): ANGLE Display::initialize error 12289: No available renderers.

This could explain problems with video playback.

You seem to be missing ntlm_auth:

0110:err:winediag:ntlm_check_version ntlm_auth was not found. Make sure that ntlm_auth >= 3.0.25 is in your path. Usually, you can find it in the winbind package of your distribution.

On Debian you can simply use apt to install it.

I'll look into the OpenVR issues. In the meantime, try yo install the quicktime codec and see what happens with GStreamer.

dasgarner commented 1 year ago

I've updated to the latest nvidia driver available and installed ubuntu-restricted-extras but I still have the missing plugin message. Vulcan isn't complaining anymore though.

I installed winbind and the ntlm_auth error has also gone.

wine_terminal_log.txt

Esinko commented 1 year ago

I've updated to the latest nvidia driver available and installed ubuntu-restricted-extras but I still have the missing plugin message. Vulcan isn't complaining anymore though.

I installed winbind and the ntlm_auth error has also gone.

wine_terminal_log.txt

Wonderful that we've at least gotten a little further. The missing plugin errors should be fixed by just installing the proper plugins. I am unfamiliar with Gstreamer, but I'd just try to install the base, ugly, bad and good plugin libraries and make sure both 32-bit and 64-bit are properly installed.

Anyway, what would happen if you tried something else than .mp4? Are the OpenVR errors still present after the driver update?

dasgarner commented 1 year ago

Wonderful that we've at least gotten a little further.

Agreed - and it is stable showing just images, which is good! 👍

Anyway, what would happen if you tried something else than .mp4?

I can't get any type of video playing. This is the command I used to install it:

apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio

I have since fixed the plugin errors though, by installing the gstreamer1.0-plugins-base:i386 package (also for good, bad and ugly). It didn't fix the video though.

Are the OpenVR errors still present after the driver update?

Yes they are, and the Cef process is still crashing.

Esinko commented 1 year ago

Wonderful that we've at least gotten a little further.

Agreed - and it is stable showing just images, which is good! 👍

Anyway, what would happen if you tried something else than .mp4?

I can't get any type of video playing. This is the command I used to install it:
apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio
I have since fixed the plugin errors though, by installing the gstreamer1.0-plugins-base:i386 package (also for good, bad and ugly). It didn't fix the video though.

Are the OpenVR errors still present after the driver update?

Yes they are, and the Cef process is still crashing.

Could I get the latest logs for trying to display browser and video content?

Esinko commented 1 year ago

Wonderful that we've at least gotten a little further.

Agreed - and it is stable showing just images, which is good! 👍

Anyway, what would happen if you tried something else than .mp4?

I can't get any type of video playing. This is the command I used to install it:
apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio
I have since fixed the plugin errors though, by installing the gstreamer1.0-plugins-base:i386 package (also for good, bad and ugly). It didn't fix the video though.

Are the OpenVR errors still present after the driver update?

Yes they are, and the Cef process is still crashing.

Do you have multiple GPUs?

dasgarner commented 1 year ago

Do you have multiple GPUs?

This machine does have two of the Quadro 4200's in it yes, although only one of those has a monitor connected to it.

Could I get the latest logs for trying to display browser and video content?

This log is for a short run of the player from start up to showing the first layout which has both a web browser and a video on it.

wine_terminal_log.txt

Esinko commented 1 year ago

This machine does have two of the Quadro 4200's in it yes, although only one of those has a monitor connected to it.

Wine's dxvk does not support this: 0184:fixme:d3d:wined3d_adapter_init_gl_caps A set of 2 devices is not supported. We need to filter out the other GPU from available adapters. I'll look for a solution for you.

Esinko commented 1 year ago

This machine does have two of the Quadro 4200's in it yes, although only one of those has a monitor connected to it.

Wine's dxvk does not support this: 0184:fixme:d3d:wined3d_adapter_init_gl_caps A set of 2 devices is not supported. We need to filter out the other GPU from available adapters. I'll look for a solution for you.

@dasgarner Could you install the vulkaninfo utility and send it's output? We might be able to setup a device filter. However, considering you are using an NVIDIA GPU, what would happen if you disable dxvk and just try to run the player?

To disable dxvk:

In order to remove DXVK from a prefix, remove the DLLs and DLL overrides (winecfg), and run wineboot -u to restore the original DLL files.

dasgarner commented 1 year ago

With dxvk disabled I get a page segfault on start.

I think we have reached a point where this isn't a good test anymore? Maybe I can try and find a different machine with only one GPU.

Esinko commented 1 year ago

With dxvk disabled I get a page segfault on start.

I think we have reached a point where this isn't a good test anymore? Maybe I can try and find a different machine with only one GPU.

Perhaps. Running with uncommon configurations, such as with 2 GPUs, can cause various issues. At least we know 2 GPUs don't seem to work at all in this case.

Esinko commented 1 year ago

UPDATE I currently have 4 machines running on Wine 8.0 with DXVK and dotnet472 displaying embedded web content and custom web based widgets. Uptime is basically 100%, crashes do happen, but I built my own restart functionality (as the daemon does not work at all). I've gotten good feedback from the client and I'm planning on open sourcing some of my code to get the player running smoothly.

Everything still pretty much boils down to the instructions in my original issue post, just with Wine 8.0.

dasgarner commented 1 year ago

That sounds encouraging! We'd be happy to run the test again with Wine 8 once you're happy on your side.

Commifreak commented 1 year ago

and we need to make a decision regarding it's future

Is there any decision made or any news?

Esinko commented 1 year ago

and we need to make a decision regarding it's future

Is there any decision made or any news?

I'm working on Dockerizing the Windows player. I've been rather busy lately, but I'm pretty much done with it.

dasgarner commented 1 year ago

and we need to make a decision regarding it's future

Is there any decision made or any news?

A player for Linux remains a popular request, but currently has to sit in the queue behind the work we're doing for Xibo v4.

We do not see a future for the C++/GTK player in this repository, but we are exploring multiple options alongside and including the work Esinko is kindly doing.

Commifreak commented 1 year ago

Now, that v4 comes closer. Are there any plans for this milestone? Any updates on the docker version? I'd love to remove the snap solution... :)

Esinko commented 1 year ago

I am indeed working on it. The largest issue I have at the moment is testing, so if someone could help with that I could publish the repository asap, maybe even today.

Commifreak commented 1 year ago

If testing main/general things is enough, I would be happy to help. We have a display next to my office, which I can observe. I would simply assign our layout, if this is enough. Or do you need special tests?

Esinko commented 1 year ago

If testing main/general things is enough, I would be happy to help. We have a display next to my office, which I can observe. I would simply assign our layout, if this is enough. Or do you need special tests?

Yeah that'll do. I have only been able to test web content so far, but I'm already aware that video playback will require some work. I'm basically asking for wine's logs when issues happen, that's all.

EDIT: I'll be able to get on this in around 8h from now.

Esinko commented 1 year ago

If testing main/general things is enough, I would be happy to help. We have a display next to my office, which I can observe. I would simply assign our layout, if this is enough. Or do you need special tests?

Yeah that'll do. I have only been able to test web content so far, but I'm already aware that video playback will require some work. I'm basically asking for wine's logs when issues happen, that's all.

EDIT: I'll be able to get on this in around 8h from now.

Here is the repository. It's not fully functional yet, but the installer runs and the player configuration pops up nicely. I have no reason to believe at least web content wouldn't work just like before.

Commifreak commented 1 year ago

Currently testing. I had several issues to build the image and made some improvements to the Dockerfile and entrypoint. As soon as it runs, I will make an PR.

Esinko commented 1 year ago

Currently testing. I had several issues to build the image and made some improvements to the Dockerfile and entrypoint. As soon as it runs, I will make an PR.

I've made fixes since my last comment. It builds fine on a few of my laptops. The player starts fine on my end, so if you're stuck try to clone the repo again. It includes fixed to both the Dockerfile and entrypoint.

Commifreak commented 1 year ago

Ah, I see. Yes exacrly the wget was silently failing. I also set the COPY to the end of the dockerfile (for testings inside entrypoint better as we dont need to build all over again, its just the entrypoint that changes).

I also added RUN chmod +x ./entrypoint.sh because the shell script is not executable when cloning.

I discard my changes and start over.

tgotowik commented 1 year ago

Hi, I just followed your initial post trying to set it up for apptainer on ubuntu 22.04. How did you handle the Mono warning on installing dotnet742. It refuses to install dotnet472 because it says that it is broken with wine-7.17.

Esinko commented 1 year ago

Hi, I just followed your initial post trying to set it up for apptainer on ubuntu 22.04. How did you handle the Mono warning on installing dotnet742. It refuses to install dotnet472 because it says that it is broken with wine-7.17.

Wine-mono should not be installed at all, it's not needed. Winetricks should automatically uninstall it when you install any dotnet package. Anyway, you can run the player in a Docker container, so you don't have to bother with installing anything yourself. Check out the repo I linked in one of my previous comments.

tgotowik commented 1 year ago

I am working with apptainer only is the problem for me. It may be possible to run a docker container with apptainer too, but I want to build it natively.

Esinko commented 1 year ago

Alright, well I'd just install the latest version of Wine. The instructions are a little outdated.

tgotowik commented 1 year ago

Okay, why did you choose v3-R301.1 and not the latest v3 R310 before the new major release?

Esinko commented 1 year ago

Okay, why did you choose v3-R301.1 and not the latest v3 R310 before the new major release?

It was the first one I happend to try that worked. No real reason. Some later versions might work.

xibosignage / xibo-linux

Linux player development should be abandoned in favor of making the Windows player more compatible with Wine #277

Short tutorial: How to run the Xibo Windows player with Wine