search

TASEmulators / BizHawk

BizHawk is a multi-system emulator written in C#. BizHawk provides nice features for casual gamers such as full screen, and joypad support in addition to full rerecording and debugging tools for all system cores.
http://tasvideos.org/BizHawk.html
Other
2.13k stars 380 forks source link

Problematic dependencies #2261

Open YoshiRulz opened 4 years ago

YoshiRulz commented 4 years ago

If it wasn't obvious, these prevent us from moving to .NET 8 (see #1415).

re: adopting existing libraries to replace our own code, Yoshi has some notes here. If there's enough interest in that you're welcome to open a tracking issue on GitHub instead.

nattthebear commented 4 years ago

Could we leave the net standard libs out of this? They're already net core compatible

I don't see us changing UI framework before .NET 5, so Windows and Linux will need separate "binaries".

We're already in the situation where the windows build and the linux build use completely different copies of system.windows.forms, yet things work with us only building binaries once. Why will that change?

nattthebear commented 4 years ago

Related https://github.com/google/flatbuffers/issues/4162

YoshiRulz commented 4 years ago

Could we leave the net standard libs out of this?

Moved out of table.

Why will [build once, run anywhere] change?

Currently, the same binary can be used on Mono and Framework on Windows because it's the same TFM (net48). Using net5;net48 will allow a binary for Windows, and a binary for Mono, both with WinForms support. net5 cannot be loaded by Mono, and Mono will never get Core 3 or .NET 5 support because the planned .NET 6 will unify the BCL implementations.

nattthebear commented 4 years ago

because the planned .NET 6 will unify the BCL implementations

Do you have a source for this? I've been looking and it's hard to find good information

YoshiRulz commented 4 years ago

[<praise of .NET Core>] .NET 5 builds on this work, taking .NET Core and the best of Mono to create a single platform that you can use for all your modern .NET code. [... .NET 5 will] expand the capabilities of .NET by taking the best of .NET Core, .NET Framework, Xamarin and Mono.

—Introducing .NET 5, 2019-05-06

Last year, we laid out our vision for one .NET and .NET 5. we said we would take .NET Core and Mono/Xamarin implementations and unify them into one base class library (BCL) and toolchain (SDK). [... Because of the pandemic,] unification will be truly completed with .NET 6, our Long-Term Support (LTS) release. Our vision hasn’t changed, but our timeline has.

—Announcing .NET 5 Preview 4 and our journey to one .NET, 2020-05-19

YoshiRulz commented 3 years ago

So I had a play with this thing (the Windows version works in Mono, who would have guessed) and the situation doesn't look too bad. GongShell, OpenTK 3, and SlimDX all reference System.Windows.Forms, but should work in .NET 6 on Windows. IIRC natt saw that in practice when testing the hammer-and-duct-tape branch. and don't seem to work in .NET 6 on Windows.

In other news, .NET 6 doesn't include WinForms for Linux, even though Mono currently supports it, and they're instead pushing ahead with MAUI (a.k.a. Xamarin.Forms). Given that, my plans for the Linux port are as follows:

edit 2022-08-12: I've just been made aware of the ActiveQt library/framework which appears to require choice of Managed C++ or COM? Not sure if Mono supports either. edit 2024-03-21: Found this thing on HN which I think is ImGui but with vectors? https://github.com/rive-app/rive-sharp

InfamousKnight commented 3 years ago

Opentk 4.4.0 came out like 5 days ago and mentions joystick fixes. We can't update that on Linux from my understanding of this?

YoshiRulz commented 3 years ago

No and that's off-topic.

Sappharad commented 3 years ago
  • Get tired of doing nothing. WinForms might not be getting removed from .NET, but it's also not getting any better. At the same time, all our library projects are stuck on Standard 2.0, unable to use some of the C# 8.0 and 9.0 language features. To reunify the binaries, we'll first need to decide if we want to switch to MAUI or to a third-party option (I'm only aware of Dear ImGui), and then focus on the one framework instead of dividing our efforts.

The other third-party option that was recommended to me several years ago was Eto, which translates to native options on their respective operating systems (WinForms, WPF, GTK, Cocoa): https://github.com/picoe/Eto The API very much resembles that of WinForms, which made it an easy choice back then. Although it seems Eto is still getting active commits, I'm not necessarily recommending it but rather just issuing a reminder that it exists.

I had started a crappy recreation of the WinForms UI in it several years ago, only to abandon it because I didn't want to spend the time necessary to recreate everything and moved back to WinForms before mostly suspending the macOS port entirely. https://github.com/Sappharad/BizHawk/tree/EtoHawk/BizHawk.Client.EtoHawk (That last commit from 2018 is actually a problem I ran into later with WinForms and ended up solving, but I don't think it affected Windows.)

YoshiRulz commented 2 years ago

SharpCompress cannot be upgraded past 0.26.0—currently tracking as adamhathcock/sharpcompress#604. fixed in ae9135807, I was overcomplicating it

tracking opentk/opentk#937 re: OTK3 keyboard init heisenbug; my OpenTK watchlist

leo60228 commented 2 years ago

FlatBuffers.Core is netstandard2.0-only, from what I can tell.

leo60228 commented 2 years ago

https://github.com/emclient/mac-playground has a port of Mono's WinForms to netstandard2.1. I'm not clear on whether it's compatible with Linux or just macOS, though.

Sappharad commented 2 years ago

That is just for macOS. It’s what I was using for the Mac port after Apple dropped 32-bit support. It won’t help you on Linux

leo60228 commented 2 years ago

OpenTK.GLControl was renamed to OpenTK.WinForms, which is in beta and supports netcoreapp3.1: https://www.nuget.org/packages/OpenTK.WinForms

jdpurcell commented 2 years ago

Potential SlimDX replacement, targets .NET Standard 2.0: https://github.com/amerkoleci/Vortice.Windows

Supports Direct3D9, DirectInput, XInput, and XAudio2 among others, but not DirectSound. A few years ago I tried out the XAudio2 part and got it working in my forked repo: https://github.com/jdpurcell/BizHawk/blob/35ef85936666dbfdc1d6ad6bf87251f95cb51dbd/BizHawk.Client.EmuHawk/Sound/Output/XAudio2SoundOutput.cs

YoshiRulz commented 2 years ago

Vortice.XAudio2 doesn't seem to have a method for enumerating devices, i.e. to replace this: https://github.com/TASEmulators/BizHawk/blob/c0056123cd198ba51408a2087d57c131f95e89a3/src/BizHawk.Bizware.DirectX/XAudio2SoundOutput.cs#L27

I also noticed SharpAudio in its description which encapsulates both XAudio2 and OpenAL, but that library is .NET Standard 2.1+ (see feliwir/SharpAudio#44). edit: and adopted Vortice recently, so the latest beta on NuGet doesn't include it edit edit: forked it, this branch is 2 tiny commits on top of the latest beta (not on NuGet) which add .NET Standard 2.0 support, and this branch has a Nix expression which should be able to build any future beta/release w/ the same patches.

edit 2023-04: Vortice.Win32 is not suitable as a replacement for the external methods in BizHawk.Common/Win32.

jdpurcell commented 2 years ago

I believe that's because Vortice uses XAudio2 2.9/2.8 whereas SlimDX uses XAudio2 2.7. Microsoft stated in the changes for XAudio2 2.8:

Device enumeration functionality has been removed from XAudio2; the GetDeviceDetails and GetDeviceCount functions have been removed from the IXAudio2 interface. Apps that want to render to other audio devices on the system must pass a device identifier string to CreateMasteringVoice instead of a device index. The default audio render device can still be created without enumeration.

They don't make it clear how to get the "device identifier string" but I found a discussion where someone mentions Windows.Devices.Enumeration.DeviceInformation.FindAllAsync. I didn't want to mess with it anymore after that 😋.

I think Direct3D9 might be the bigger problem though - I did start trying to convert some of that code out of curiosity, and although I'm not very familiar with Direct3D, I'm pretty sure Vortice's implementation of D3D9 is incomplete. My impression is that the author would be unwilling to fix that himself, although would accept pull requests if someone wanted to finish the missing features.

YoshiRulz commented 2 years ago

I have a feeling we stumbled onto the same social.MSDN thread (login required)... edit: found another C++ code snippet that doesn't require login

We won't have any problems w/ D3D9 if we switch to its D3D12 backend instead. (And if we use Vortice via Veldrid then Vulkan is also available. Verily.)

YoshiRulz commented 2 years ago

This seems a good place to put this—.NET Framework 4.8.1 is real and available now, but only for Win10+, and it only includes the promised accessibility changes, not .NET Standard 2.1 support or anything "new". I don't think we should upgrade, and I'm not even going to bother checking if Mono runtime supports it.

CasualPokePlayer commented 2 years ago

FlatBuffers.Core ? Unknown (Emulation.Cores). IDK, natt hacked this in for Nyma. Presumably serialisation.

Seems like since https://github.com/TASEmulators/BizHawk/commit/b56e229a45446fa62026ab16aeab201b26104bf4 our copy targets NET Standard 2.0, so shouldn't actually be any issues here with that I think.

Related google/flatbuffers#4162

Seems that this too has moved along (official NuGet available now, albeit still beta), probably not really needed for this issue though?

YoshiRulz commented 2 years ago

Saw that. I will be switching to NuGet once it's stable, even if it's not strictly necessary.

YoshiRulz commented 1 year ago

@nattthebear I've pushed branch update-flatbuffers to do that ^ but I believe this change means that the C++ flatbuffers.h needs updating also. https://github.com/TASEmulators/BizHawk/blob/e87d3065718719b2abd5895886dceeb91d7f7346/waterbox/nyma/NymaTypes_generated.h#L11-L14

nattthebear commented 1 year ago

Seems reasonable. You should just be able to delete the contents of waterbox/nyma/common/flatbuffers/* and replace with a new version of flatbuffers, and then rebuild all the nyma cores.

YoshiRulz commented 1 year ago

I still can't do that on my machine. With nattthebear/musl#3 I get through most of it until:

$> ./make-all-released-cores.sh
cxx mednafen/src/error.cpp
cxx mednafen/src/VirtualFS.cpp
cxx mednafen/src/FileStream.cpp
cxx mednafen/src/MemoryStream.cpp
cxx mednafen/src/Stream.cpp
cxx mednafen/src/file.cpp
In file included from mednafen/src/mednafen.h:4,
                 from mednafen/src/error.cpp:22:
mednafen/src/types.h:58:10: fatal error: cmath: No such file or directory
   58 | #include <cmath>
      |          ^~~~~~~
In file included from mednafen/src/mednafen.h:4,
                 from mednafen/src/VirtualFS.cpp:33:
mednafen/src/types.h:58:10: fatal error: cmath: No such file or directory
   58 | #include <cmath>
      |          ^~~~~~~
compilation terminated.
[...]

I've force pushed the branch for someone else to check. I only overwrote the flatbuffers headers that were present, assuming the rest were unused. Added the rest to resolve below problem. Can those come from a submodule instead?

CasualPokePlayer commented 1 year ago

I tried building them on your branch myself and ran into a different error, with "flatbuffers/array.h" missing. Seems there's also a lot of other header files missing too.

The missing <cmath> error anyways I think is just something went wrong for you when you built the llvm/libcxx part (it should normally reside in sysroot/include/c++/v1/cmath

YoshiRulz commented 1 year ago

Pushed https://github.com/TASEmulators/BizHawk/commit/04fcf59afe75e7b4e938ea1ad0f10ab5f467529a and 3 prior commits.

alkaris2 commented 1 year ago

Seems I got the issue with the System.Windows.Forms thing.

BizHawk/src/BizHawk.Client.EmuHawk/MainForm.cs(3434,39): error CS1503: Argument 2: cannot convert from 'BizHawk.Client.EmuHawk.FolderBrowserEx' to 'System.Windows.Forms.CommonDialog'

My current version of .NET on Linux is 7.0, SDK Version: 7.0.101, is there no work around for this to make it work? Can we just skip the System.Windows.Forms.CommonDialog and make it use the version in 7.0 and build anyway?

CasualPokePlayer commented 1 year ago

The error is something wrong from the most recent commit (affects Windows too). Just go back to the previous commit or wait until it's fixed. This is fixed.

CasualPokePlayer commented 1 year ago

Something to note, OpenTK.Input 4 is actually netstandard 2.0 (the rest of OpenTK is not), so us using it for host input would be (mostly) a non-blocker (host graphics/sound is obviously still an issue).

YoshiRulz commented 1 year ago

There was already an option for input on Linux, evdev-sharp, which would have involved building from source. But while we're on the subject, allow me to quote myself:

OpenTK 4 and 5 are .NET Core only, though IIRC OTK 4 was .NET Standard until late in beta, maybe it can still work just by adding the target and building from source?

CasualPokePlayer commented 1 year ago

Regardless, using OpenTK.Input means no building from source, the latest nuget package is already netstandard2.0 compatible.

If we want another input backend, something to look into with Silk.NET is its SDL2 Input bindings, which may be nice in any case due to SDL2 supporting a ton of different controllers.

CasualPokePlayer commented 1 year ago

Doing some work on the input, it seems more complicated when I thought. OpenTK.Input in 4.x is not the same as OpenTK.Input in 3.x, as it is supposed to be some "Hid/Controller" code (in reality, there's barely anything in there and looks to be just incomplete all in all). The old OpenTK input code was moved to OpenTK.Windowing.GraphicsLibraryFramework, and has a significantly different API. Differently enough that I'm not sure we can use it anymore. The main issue is simply that it requires that an OpenTK NativeWindow be created, and that window be focused, in order for it to obtain any input. This obviously is a bit not good, as obviously the MainForm is what we'd want focused (and since it has to be an OpenTK NativeWindow, we can't give it the MainForm anyways), and we would lose the ability to have input without focus like with OpenTK 3.x (which is probably important too for the context of controller dialog, where that might not have MainForm neccessarily focused in this context? maybe not, not sure).

(sidenote: OpenTK 4.x internally just uses GLFW to handle getting input in a platform agnostic way, while OpenTK 3.x just used its own platform abstractions to native APIs)

SDL2 similarly has this issue, but it does have an option to create a window from an existing window using a native window handle (which is handed over in the first place by our input API). Even then however, that doesn't work entirely, at least on Windows (maybe on Linux too, dunno), as for whatever reason as WM_KEYDOWN/WM_KEYUP just never arrive, despite the MainForm clearly having focus, so keyboard input just doesn't work at all.

Not entirely sure what'd be a good solution here. We could use evdev-sharp as suggested, but since that's just evdev that will only work for Linux, and not any future macOS/whatever port and would mean Windows users are stuck with DirectInput+XInput (which is fine mostly, but it is limiting and options like SDL2 would provide more options in this respect). (Not that this doesn't mean evdev-sharp shouldn't be a future option, but a catch-all works on all OSes would be beneficial for us to have first).

YoshiRulz commented 1 year ago

a catch-all works on all OSes would be beneficial for us to have first

I'm increasingly souring on this position. macOS all but precludes that anyway (as does Windows UWP I believe).

CasualPokePlayer commented 1 year ago

Continuing to look and it really just seems like we will forced to just write OS specific code for each platform for keyboard input. We can have SDL2 at least maybe as a nice catch all for gamepads (doesn't have dumb focused window requirement), but that's it. At least it seems possible across all platforms.

I currently envision this for input:

Windows: DirectInput+XInput, DirectInput+SDL2 Linux: evdev, evdev+SDL2 macOS: Quartz+SDL2

On that, this isn't perfect. We may need to change the keyboard from using DirectInput, for UWP support, as UWP does not support DirectInput, the only alternative here seems to be using the RAWINPUT API (as a bonus, we can just make DirectInput back to no longer a hard dependency this way, as the removal of OpenTK would implicitly do for Windows users). Linux doesn't necessarily have to use evdev for keyboards, X11 and libinput can work, although evdev probably would be better as a nice catch all for Linux (and BSDs?). macOS lol find someone who cares enough for some OS specific macOS code, otherwise that would stay with SDL2 for gamepad support (and I'll write up the keyboard support here).

On another note here, I don't like how the main form handle is passed on to input APIs, especially with our move to a new gui backend not making that a possibly not particularly possible idea (it is predicated on able to obtain that handle and being able to hook onto the window proc for reiniting gamepads for directinput), it would function the same if it created a hidden window internally (nb: also this would be needed for RAWINPUT use, as it responds to WM_INPUT messages), and that would be easy enough to message pump that new native window on some new thread...

EDIT: RAWINPUT handler is done and used with the SDL2 backend (also DirectInput actually wraps around RAWINPUT internally anyways, lol)

YoshiRulz commented 5 months ago

Possibly not the right thread for this: https://github.com/meziantou/Meziantou.Polyfill is similar to PolySharp, but the list of backports is different. Most relevant is that it includes some extension methods. (PolySharp seems to eschew methods added to existing types—which is fair enough, since the call must be to a nonstandard type when polyfilled i.e. not binary-compatible. But we needn't care about that for our internal libraries, only whether the combined app works.) Naively adding the dependency in results in a build failure because both trigger for the same polyfills (I'm pretty sure they run in series and the second could detect the first...), but both are also configurable. If it can reduce the amount of backports we have to maintain then I say it's worth a try.