Update Dolphin to upstream

[JesseTG]

This isn't ready to merge yet; it compiles, but I'm getting tons of runtime errors that I'm still figuring out.

[srk15372]

cool thanks, hope you could achieve this

[antoniodesousa]

This is great! I was just planning to do the same hahaha

[Gamr13]

Christ mate, fair play!

[JesseTG]

Thanks for the moral support! Nonetheless, I'm facing some challenges.

So far, the major problems I'm facing are as follows.

The Dolphin core supports three video drivers, and they're all broken in slightly different ways. Observe:

Then there's whatever the hell this is:

Even if I ignore the video driver errors, I still get "Invalid read" errors, regardless of whether the CPU is emulated by interpreter or by JIT. @hunterk suggested that the upstream build might have broken PIC compatibility, but I don't think that's the case (otherwise the interpreter would work okay).

I'm not familiar enough with Dolphin to be able to resolve this on my own. @aliaspider is MIA, so is anyone else able to provide me leads?

[antoniodesousa]

I wouldn't know out of my head. I'm also not super familiar with Dolphin. Specially, the upstream branch. If I had to guess, it's probably something wrong in the DolphinLibretro module. You could also try to contact the official Dolphin team about but I heard that they are not really in good terms with the libretro team. I really hope that you succeed on this though. I have been waiting for someone to update this branch for awhile now.

I probably can help you debug the runtime errors if you can help me how to compile the code. I get a bunch of errors everytime I tried.

[kira1000]

out of curiosity, what GPU do you have? I did a bit of digging on that shader error and people on the Dolphin forum are saying it's because of an old GPU. Would also help to know which error is with which video driver, as some people have suggested trying Direct3D 11 (this was 4 years ago on the dolphinemulator subreddit).

Another thought, are you using a linux distro by any chance? I could test it under windows to see if that does anything, since windows fonts are mentioned and in one of the posts on the Dolphin forum the error was appearing on an Ubuntu build.

[JesseTG]

I wouldn't know out of my head. I'm also not super familiar with Dolphin. Specially, the upstream branch. If I had to guess, it's probably something wrong in the DolphinLibretro module. You could also try to contact the official Dolphin team about but I heard that they are not really in good terms with the libretro team. I really hope that you succeed on this though. I have been waiting for someone to update this branch for awhile now.

I probably can help you debug the runtime errors if you can help me how to compile the code. I get a bunch of errors everytime I tried.

@antoniodesousa Could you tell me the following:

• What OS you're using to compile this PR
• What compiler you're using
• How you're invoking CMake (this includes any options you set)

out of curiosity, what GPU do you have? I did a bit of digging on that shader error and people on the Dolphin forum are saying it's because of an old GPU.

A NVIDIA GeForce RTX 3060. I wouldn't call it old with a straight face.

Would also help to know which error is with which video driver, as some people have suggested trying Direct3D 11 (this was 4 years ago on the dolphinemulator subreddit).

vulkan, d3d11, and gl, which are the drivers that the libretro core supports. I haven't tried any others because the libretro core doesn't support them, and adding such support is beyond the scope of this PR. Software rendering's broken too.

Another thought, are you using a linux distro by any chance?

No, I'm on Windows 10. I do use WSL regularly, but my primary dev efforts are on the Windows host.

[antoniodesousa]

@JesseTG I managed to compiled it in the weekend. I will take a look when I have time. Thanks, anyways!

How's the progress so far?

[JesseTG]

Stuck until I figure out these runtime errors. I've been working on something else in the meantime. Anything you can uncover will be immensely helpful.

[kira1000]

Hm…what game/file format are you using? If it’s nkit they can be buggy.

[antoniodesousa]

@JesseTG I found something that's not right. Your fork is missing quite a few submodules compared to the upstream. This is the list of submodules from your fork:

And this is the upstream:

[JesseTG]

Are you on the right branch? The jtg/update-dolphin branch includes those upstream submodules.

[JesseTG]

Hm…what game/file format are you using? If it’s nkit they can be buggy.

F-Zero GX, in .rvz format.

[antoniodesousa]

@JesseTG Ah my bad... wrong branch indeed

[JesseTG]

No worries, it happens.

[kira1000]

got some info for you, apparently the most likely cause of the issue is the sys folder not being properly populated, which is where Dolphin gets essential files.

EDIT: I am afraid it may be a lost cause-apparently the core is fundamentally broken, according to the Dolphin dev team. Heck it relied on dualcore to render AT ALL (which may explain the issues).

[JesseTG]

got some info for you, apparently the most likely cause of the issue is the sys folder not being properly populated, which is where Dolphin gets essential files.

I have since figured this part out, thank you.

EDIT: I am afraid it may be a lost cause-apparently the core is fundamentally broken, according to the Dolphin dev team. Heck it relied on dualcore to render AT ALL (which may explain the issues).

I'm not entirely sure that the outlook is this bleak.

If the issue is the core game loop, what if we could break the loop into smaller pieces? And then reuse those pieces in a slightly different way that fits the needs of libretro? It would be like buying a Lego set and using most of the pieces, but also building something that's a little different than what's on the box. The goal here would be to keep the core's use of threads on our terms.

[antoniodesousa]

The reason why the Dolphin core doesn't support single core, it's because the code was based in a release of the upstream where it was broken there as well. They fixed it in later versions but it was never backported to libretro.

You can read more about it here @JesseTG Anyways, I don't think this is the reason why you get so many runtime errors.

[hizzlekizzle]

I don't think the dual-core thing is directly related to the upstream issue from many years ago. The previous rebase used a clean upstream snapshot from the time, so it's a couple of years old by now but not that many

[antoniodesousa]

@JesseTG I was trying to build your branch today, but I was getting several errors. Then I compared it with the upstream (https://github.com/dolphin-emu/dolphin), I see that 2 submodules are missing?

Externals/cubeb and Externals/libusb

[JesseTG]

@JesseTG I was trying to build your branch today, but I was getting several errors. Then I compared it with the upstream (https://github.com/dolphin-emu/dolphin), I see that 2 submodules are missing?

Externals/cubeb and Externals/libusb

Some of these submodules have their own submodules, and I think these two are offenders. Try this StackOverflow post.

While we're at it, how exactly did you clone the repo? Or, if you initialized submodules after cloning, how exactly did you do so?

[antoniodesousa]

git clone https://github.com/JesseTG/dolphin.git git switch jtg/update-dolphin git submodule update --init --recursive

[antoniodesousa]

@JesseTG I finally managed to successfully compiled your branch. I had to add Externals/cubeb and Externals/libusb myself though... Anyways, I kinda managed to have F-Zero at least running. I can only hear the sound and inputs seem to work, but no video output whatsoever. Just add this line to .\Source\Core\DolphinLibretro\Boot.cpp

Config::SetBase(Config::MAIN_USE_PANIC_HANDLERS, false);

I'm pretty sure the shader compilation is broken in the libretro project that's why there's no video output. The Dolphin team made a lot of changes to the shader compilation in the last years and none of those commits were backported. So, I guess the core is not longer compatible with those changes.

I would love to help but I have no idea how to rewrite the libretro project to fix it. You need someone with experience with the Dolphin APIs that can help...

[JesseTG]

@JesseTG I finally managed to successfully compiled your branch. I had to add Externals/cubeb and Externals/libusb myself though... Anyways, I kinda managed to have F-Zero at least running. I can only hear the sound and inputs seem to work, but no video output whatsoever. Just add this line to .\Source\Core\DolphinLibretro\Boot.cpp

Config::SetBase(Config::MAIN_USE_PANIC_HANDLERS, false);

If you've confirmed that sound and input still work, that's already a huge step. You've helped me out immensely; thank you so much.

I'm pretty sure the shader compilation is broken in the libretro project that's why there's no video output. The Dolphin team made a lot of changes to the shader compilation in the last years and none of those commits were backported. So, I guess the core is not longer compatible with those changes.

Those changes should've been merged, but I might not have done so properly. Or those changes involve code that the core isn't currently calling.

[antoniodesousa]

Those changes should've been merged, but I might not have done so properly. Or those changes involve code that the core isn't currently calling.

Sorry, that I wasn't clear enough. I meant that the changes that the Dolphin team did were not reimplemented in the libretro project (DolphinLibretro). Anyways, I really hope that you can figure out why the shader compilation is currently broken.

And Merry Christmas!!

[JesseTG]

Thanks, you too!

[j8r]

I tried to compile on Linux, got:

``` In file included from /usr/include/X11/Xlib.h:44, from /dir/Externals/Vulkan/Include/vulkan/vulkan.h:59, from /dir/Source/Core/VideoBackends/Vulkan/VulkanLoader.h:24, from /dir/Source/Core/DolphinLibretro/Video.h:12, from /dir/Source/Core/DolphinLibretro/Video.cpp:2: /dir/Source/Core/Core/NetworkCaptureLogger.h:31:3: error: expected identifier before numeric constant 31 | None, | ^~~~ /dir/Source/Core/Core/NetworkCaptureLogger.h:31:3: error: expected '}' before numeric constant In file included from /dir/Source/Core/Core/Debugger/PPCDebugInterface.h:13, from /dir/Source/Core/Core/PowerPC/PowerPC.h:15, from /dir/Source/Core/DolphinLibretro/Options.h:9, from /dir/Source/Core/DolphinLibretro/Video.cpp:31: /dir/Source/Core/Core/NetworkCaptureLogger.h:30:1: note: to match this '{' 30 | { | ^ /dir/Source/Core/Core/NetworkCaptureLogger.h:31:3: error: expected unqualified-id before numeric constant 31 | None, | ^~~~ /dir/Source/Core/Core/NetworkCaptureLogger.h:56:11: error: 'NetworkCaptureType' does not name a type; did you mean 'NetworkCaptureLogger'? 56 | virtual NetworkCaptureType GetCaptureType() const = 0; | ^~~~~~~~~~~~~~~~~~ | NetworkCaptureLogger /dir/Source/Core/Core/NetworkCaptureLogger.h:72:3: error: 'NetworkCaptureType' does not name a type; did you mean 'NetworkCaptureLogger'? 72 | NetworkCaptureType GetCaptureType() const override; | ^~~~~~~~~~~~~~~~~~ | NetworkCaptureLogger /dir/Source/Core/Core/NetworkCaptureLogger.h:81:3: error: 'NetworkCaptureType' does not name a type; did you mean 'NetworkCaptureLogger'? 81 | NetworkCaptureType GetCaptureType() const override; | ^~~~~~~~~~~~~~~~~~ | NetworkCaptureLogger /dir/Source/Core/Core/NetworkCaptureLogger.h:100:3: error: 'NetworkCaptureType' does not name a type; did you mean 'NetworkCaptureLogger'? 100 | NetworkCaptureType GetCaptureType() const override; | ^~~~~~~~~~~~~~~~~~ | NetworkCaptureLogger /dir/Source/Core/Core/NetworkCaptureLogger.h:117:1: error: expected declaration before '}' token 117 | } // namespace Core | ^ /dir/Source/Core/Core/Debugger/PPCDebugInterface.h:87:25: error: 'NetworkCaptureLogger' is not a member of 'Core'; did you mean 'NetworkCaptureType'? 87 | std::shared_ptr NetworkLogger(); | ^~~~~~~~~~~~~~~~~~~~ | NetworkCaptureType /dir/Source/Core/Core/Debugger/PPCDebugInterface.h:87:45: error: template argument 1 is invalid 87 | std::shared_ptr NetworkLogger(); | ^ /dir/Source/Core/Core/Debugger/PPCDebugInterface.h:87:8: error: '' in namespace 'std' does not name a type 87 | std::shared_ptr NetworkLogger(); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /dir/Source/Core/Core/Debugger/PPCDebugInterface.h:94:25: error: 'NetworkCaptureLogger' is not a member of 'Core'; did you mean 'NetworkCaptureType'? 94 | std::shared_ptr m_network_logger; | ^~~~~~~~~~~~~~~~~~~~ | NetworkCaptureType /dir/Source/Core/Core/Debugger/PPCDebugInterface.h:94:45: error: template argument 1 is invalid 94 | std::shared_ptr m_network_logger; | ^ /dir/Source/Core/Core/Debugger/PPCDebugInterface.h:94:8: error: '' in namespace 'std' does not name a type 94 | std::shared_ptr m_network_logger; | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ make[2]: *** [Source/Core/DolphinLibretro/CMakeFiles/dolphin_libretro.dir/build.make:160: Source/Core/DolphinLibretro/CMakeFiles/dolphin_libretro.dir/Video.cpp.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:2334: Source/Core/DolphinLibretro/CMakeFiles/dolphin_libretro.dir/all] Error 2 make: *** [Makefile:156: all] Error 2 ```

To reproduce it in a Docker container, after running docker run --pull always -it --rm -v $PWD:/dir -w /dir docker.io/debian:

apt update
apt install cmake g++ libbluetooth-dev libegl-dev libudev-dev libx11-dev libxi-dev libgl1-mesa-dev libglu1-mesa-dev libxrandr-dev pkg-config zlib1g-dev

mkdir build
cd build
cmake -DENABLE_EVDEV=OFF -DENABLE_NOGUI=OFF -DLINUX_LOCAL_DEV=ON -DLIBRETRO=ON ..
make -j$(nproc)

[washingleavesandthensome23-434]

Any progress?, would be nice to be up to date with current nightly

[yudhiyou]

I hope this project is succeeded.

[JesseTG]

Any progress?, would be nice to be up to date with current nightly

Not yet, I'm going back and forth between several different libretro-related projects.

[ChristopherCapito]

I am a webdev so i have little to no xp as to whatever is needed to make this work but i wanted to play Sims 2 with GF so i stumbled upon this. for what its worth: great work so far :D brought back a lot of memories!

if you find the time to continue working on this that would be very cool :D

[JesseTG]

Thank you, I appreciate the sentiment!

My current focus right now is on melonDS DS (and on finding a day job); that core uses the original emulator as a dependency rather than as a fork, which means it should be very easy to maintain. I intend to give Dolphin that treatment one day, as I believe it would be better in the long run. But not before I actually finish (the hardest parts of) melonDS DS.

For now I will close this PR, because I don't see it going anywhere. At one point I'll start working on Dolphin: The Twin Snakes (title pending). When I do, I will probably start talking about it a lot.

[i30817]

A core that is just a dolphin launcher already exists. It's hampered by the dolphin version of its command line specific main not having all of the shortcuts mapped and the other usual limitations of a core using a launcher instead of real integration. I've coded on something like that before but it's a real pain not having feedback on the menu for minimal setup things even after you hookup the missing hotkeys -most notably, savestates- and forget games using the emulated wiimote. Any changes would have to restart the emulator, and I don't know of a mechanism to avoid that.

If you go down that path, you'll probably have to contribute to upstream to make the cmd line interface more than a unconfigurable toy or find a way to model the options in the config file and always feed one general\pergame to dolphin.. and that still needs upstream work because as I mentioned a lot of even the config file stuff wasn't hooked up to the dolphin cmd line main.

I gave up myself around then, because it would require to contribute upstream on a language I'm just terrible at, for something rarely used and bitrotten.