search

xenia-project / xenia

Xbox 360 Emulator Research Project
https://xenia.jp
Other
8.26k stars 1.14k forks source link

Linux Support #1430

Open bwrsandman opened 5 years ago

bwrsandman commented 5 years ago

Current State

Before

Xenia on Linux compiles for Travis tests in order to check C++ errors, style and very basic functionality. It has some nice abstract function definition. Most of these definitions are stubs with failing asserts which mean running Xenia in debug causes SIGABRT and running in release will eventually crash due to missing implementation.

The biggest missing part is the threading. Without threading implementation, the Xenia UI can't run and wait. The emulated threads and synchronization primitives which are built on top of these cannot work properly.

There is also PPC CPU emulation discrepancies between Windows and Linux. The registers used on 64-bit Windows and 64-bit Linux are not the same, therefore proper care must be made to use the correct registers when transitioning from guest to host code and vice-versa.

Memory mapping needs a bit of work due to the differences in shared memory and explicit memory ranges having different parameters and prefixes on Linux.

The GTK windowing and its interaction with Vulkan needs to fixed for several bugs.

The use of paths and the string functions need to better specify NT-style line endings (\) which are used by guest and Windows host. In the case of a Linux host, Unix line endings must be used when specifying guest paths.

Linux debugging and stack walking functions need to be implemented.

Approach to Adding Support

My approach to fixing the following problem has been to first add unit tests (and to use the ones already there) to the already working Windows implementation in order to have a behavioural ground truth. Then I add implementation to Linux which satisfy these tests. Once the Linux implementation pass the tests, I activate those tests on Travis in order to prevent future regressions.

Active PRs

Here are the PRs that I have worked on to have native Linux working. Some of these are built on top of the work of others and I preserved the authorship in the commits. They are in descending order of importance.

Things left to do

If anyone feels like contributing to the port there are a few areas which have not yet been addressed:

New State

After @JoelLinn Input

The fixes allow for all unit tests in the project to run and pass on Linux with clang.

The Xenia UI works and can select files with the open dialog.

Loading a xex will work up until emulation.

Calling a host function works as well as host calling a guest function.

Logging works.

Loading a ROM will eventually crash during emulation.

To try all the changes together:

Prism019 commented 5 years ago

linux_stack_walker has a merge conflict in .gdbinit... should probably use -X theirs for that merge?

bwrsandman commented 5 years ago

The .gdbinit conflict has to be resolved without an -X strategy. Unfortunately, using -X theirs will replace the changes from linux_threading and we need both changes from linux_threading (gdb ignoring custom signals from the threading implementation) and those from linux_stack_walker (gdb ignoring custom signal for stack retrieval). The correct strategy would be -X union but git doesn't currently support that for merge yet.

You can add this to the commands to auto-resolve in a union-like way:

$ sed -i '/^<<<<<<</d;/^=======/d;/^>>>>>>>/d' .gdbinit
$ git add .gdbinit
$ git commit --no-edit
bwrsandman commented 5 years ago

Oh yeah, I should add: Use gdb for debugging, lldb is painfully slow and I'm not sure why.

JoelLinn commented 5 years ago

What is the current state of these merge requests? There seem to be only minor objections from the maintainers. This work is the basis for all further Linux development.

Prism019 commented 5 years ago

Games still do not run. There are also some threading issues (race conditions?) with exiting the emulator. There's also the case of wide chars being different sizes between the guest and host (16 bit vs 32 bit, respectively).

bwrsandman commented 5 years ago

I wasn't aware of race conditions...

JoelLinn commented 5 years ago

To my knowledge merging this does not break anything, it only improves things. It would make it easier for others to submit further fixes for linux - doing so right now would probably make them dependent on these PRs anyways.

JoelLinn commented 5 years ago

Audio

A cross platform audio back-end is implemented in PR #1498. It is however untested on Linux (since emulation crashes at the time of writing). It was way easier than expected to implement this.

This comment was edited since my previous thoughts are no longer relevant

JoelLinn commented 5 years ago

Input

Input has been implemented in PR #1493

bwrsandman commented 5 years ago

Great work!

bwrsandman commented 5 years ago

The window title issue was fixed https://github.com/xenia-project/xenia/pull/1433/commits/0fdbcdeb35f066f50e79a848c79eb397b910ea9b It wasn't related to the different width of wchar. Xenia gets the title correctly from the game file. It was just the printf token which was using regular strings instead of wide strings.

JoelLinn commented 5 years ago

Is this it? I would argue there must be a number of places where sizeof(wchar_t) == 2 is assumed. E.g. XBox buffers converted to c++ (w)strings and vice versa.

bwrsandman commented 5 years ago

Perhaps in src/xenia/kernel/util/shim_utils.h but from my tests running this app, it's only ansi strings being called.

JoelLinn commented 5 years ago

Ok a quick static analysis on which code segments assumewchar_t on the host and uint16_t of the guest to be equal: (May not be complete)

https://github.com/xenia-project/xenia/blob/d4a5a57867b3bc8e0337b4981884e8983b4af14c/src/xenia/base/memory.h#L283-L295

https://github.com/xenia-project/xenia/blob/d4a5a57867b3bc8e0337b4981884e8983b4af14c/src/xenia/base/memory.h#L402-L408

https://github.com/xenia-project/xenia/blob/d4a5a57867b3bc8e0337b4981884e8983b4af14c/src/xenia/kernel/util/shim_utils.h#L104-L122

What about using

std::wstring_convert<std::codecvt_utf16<char32_t>, char32_t>

in the case of 4 byte wchar_t platforms to translate between host and guest?

Another way would be to use std::u16string all around - That would however introduce other problems I guess.

bwrsandman commented 5 years ago

I went and ran with std::u16string everywhere. No success. It seems to crash at the same place. 0eba293cd81ca85a3e8e00024d444862575f897c

Prism019 commented 5 years ago

That happened to me too on my local branch when i did the conversion. Trying to debug this in GDB is a nightmare though.. when I try to run dolphin.xex, the stack is clobbered when it crashes, so I don't know what to look at in IDA to try and figure out the crash.

JoelLinn commented 5 years ago

I went and ran with std::u16string everywhere. No success. It seems to crash at the same place. 0eba293

Eww... Makes me wonder what the places are where std:wstrings interact with Linux APIs Maybe use 4 byte in xenia and convert to 1 byte UTF8 when needed. We would need to do that anyways when using 2 byte strings in xenia. Downside is one more conversion between xenia and guest (4 byte <> 2 byte).

Generally speaking, a clean solution on Linux is to use std::string with UTF8 encoding but that gets pretty messy when you have ansistrings around that are also 1 byte long like we have.

Prism019 commented 4 years ago

For anyone not following along on the #dev-linux channel on the discord, I've gotten the game emulation to be stable. I'm going to check to make sure the fixes don't interfere with Windows, then open a new PR to get the changes merged.

Note: Vulkan Graphics and SDL Audio do not work as of now. SDL Input may work but I have no way of testing it in a game.

Prism019 commented 4 years ago

1517 should fix the segfaulting while running emulations.

bwrsandman commented 4 years ago

There are some conflicts with #1339 I don't know if you tested with both, I'm applying #1517 after #1339 with the -X theirs strategy. What are your thoughts?

bwrsandman commented 4 years ago

There seems to be a bug in the auto reset events in the threading PR. The notify one signal does not seem to unblock the main thread causing a deadlock. More tests need to be made to have the behaviour on par with windows. See the threading PR for more details.

sl1pkn07 commented 4 years ago

seems conflict #1339 with #1397 and #1405 with #1317

greetings

bwrsandman commented 4 years ago

Rebased and fixed the conflicts. I did minimal testing as I don't have a lot of time to dedicate to this.

Shoegzer commented 3 years ago

Thanks to PR #1792, Xenia finally builds on a local linux test system. While that's great progress, the resulting (debug) build predictably crashes, as mentioned above and shown here. Tested on Linux Mint 20.1 / kernel 5.11 with commit 95031d94713206e450dd7beabcad45b9e99d77fa.

I can do some testing with this PR to help out, but agree with @JoelLinn that it may be best to merge as-is, especially given the lack of activity in over a year at this point. Now that Linux builds are compiling in master, I can imagine devs and testers alike could produce more useful results without a need to be aware of or dependent upon this PR. I can only assume the checkoff lists mentioned in the summary are still up-to-date.

Thoughts?

SerraraFluttershy commented 3 years ago

Thanks to PR #1792, Xenia finally builds on a local linux test system. While that's great progress, the resulting (debug) build predictably crashes, as mentioned above and shown here. Tested on Linux Mint 20.1 / kernel 5.11 with commit 95031d9.

I can do some testing with this PR to help out, but agree with @JoelLinn that it may be best to merge as-is, especially given the lack of activity in over a year at this point. Now that Linux builds are compiling in master, I can imagine devs and testers alike could produce more useful results without a need to be aware of or dependent upon this PR. I can only assume the checkoff lists mentioned in the summary are still up-to-date.

Thoughts?

I concur with this comment. As a frequent user of this software (for playing games and high-level reverse engineering of said games) I also recommend all non-problematic pending commits be merged into the master branch so that more pressing matters (such as Vulkan rendering and windowing) can be addressed on the journey to a fully functional Linux release.

Should I try compiling a custom build of the software myself (incorporating the pending commits of course) and relay the results of doing so back to you all?

JoelLinn commented 3 years ago

Thanks to PR #1792, Xenia finally builds on a local linux test system. While that's great progress, the resulting (debug) build predictably crashes, as mentioned above and shown here. Tested on Linux Mint 20.1 / kernel 5.11 with commit 95031d9. I can do some testing with this PR to help out, but agree with @JoelLinn that it may be best to merge as-is, especially given the lack of activity in over a year at this point. Now that Linux builds are compiling in master, I can imagine devs and testers alike could produce more useful results without a need to be aware of or dependent upon this PR. I can only assume the checkoff lists mentioned in the summary are still up-to-date. Thoughts?

I concur with this comment. As a frequent user of this software (for playing games and high-level reverse engineering of said games) I also recommend all non-problematic pending commits be merged into the master branch so that more pressing matters (such as Vulkan rendering and windowing) can be addressed on the journey to a fully functional Linux release.

Should I try compiling a custom build of the software myself (incorporating the pending commits of course) and relay the results of doing so back to you all?

I think biggest hesitation for merging is with #1339 I think. It modifies code that is shared among platforms and has some open discussions. IIrc stuff like windowing changes are independent to CPU

sl1pkn07 commented 3 years ago

Hi

for me, without any patch or applied any PR, fail build in my archlinux (own package)

./src/xenia/base/filesystem_posix.cc:48:3: error: ignoring return value of function declared with 'warn_unused_result' attribute [-Werror,-Wunused-result]
  readlink("/proc/self/exe", buff, FILENAME_MAX);
  ^~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[1]: *** [xenia-base.make:268: obj/Linux/Release/Linux/Release/xenia-base/filesystem_posix.o] Error 1
make[1]: *** Se espera a que terminen otras tareas....
../src/xenia/base/system_linux.cc:22:10: fatal error: 'third_party/SDL2/include/SDL.h' file not found
#include "third_party/SDL2/include/SDL.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[1]: *** [xenia-base.make:307: obj/Linux/Release/Linux/Release/xenia-base/system_linux.o] Error 1
../src/xenia/base/memory_posix.cc:136:3: error: ignoring return value of function declared with 'warn_unused_result' attribute [-Werror,-Wunused-result]
  ftruncate64(ret, length);
  ^~~~~~~~~~~ ~~~~~~~~~~~
1 error generated.
make[1]: *** [xenia-base.make:289: obj/Linux/Release/Linux/Release/xenia-base/memory_posix.o] Error 1
make: *** [Makefile:290: xenia-base] Error 2
make: se sale del directorio '/tmp/makepkg/sl1-xenia-git/src/xenia/build'

ERROR: build failed with one or more errors.

commit used: 4861022158ff02bf51660dcf1dbb33eb1ad4cded

clang 12.0.1 kernel 5.13.8-arch1-1

1339, , #1405, #1431, #1433 needs rebase due conflict files

EDIT: fixed with

diff --git a/premake5.lua b/premake5.lua
index e44f81c71..463264402 100644
--- a/premake5.lua
+++ b/premake5.lua
@@ -131,7 +131,8 @@ filter({"platforms:Linux", "toolset:gcc"})

 filter({"platforms:Linux", "language:C++", "toolset:clang"})
   disablewarnings({
-    "deprecated-register"
+    "deprecated-register",
+    "unused-result"
   })
 filter({"platforms:Linux", "language:C++", "toolset:clang", "files:*.cc or *.cpp"})
   buildoptions({

greetings

sl1pkn07 commented 3 years ago

seems #1076 needs to be added to the list. with this PR now i can able to run xenia without segfaults and initated properly the vulkan subsystem on my linux

greetings

Triang3l commented 3 years ago

Yes, #1076 is an even more advanced and comprehensive version of #1431 conceptually.

CtrlC-Root commented 1 year ago

https://github.com/xenia-project/xenia/pull/2018 seems to supersede https://github.com/xenia-project/xenia/pull/1339

xiota commented 1 year ago

Need to add #include <cstdint> to

https://github.com/xenia-project/xenia/blob/00aba94b98b1369d604f233fcd6a9dbbc233f3bb/src/xenia/base/utf8.cc#L12-L16

Or change uint32_t to std::uint32_t ?

illusion0001 commented 1 year ago

Fixed in #2211 I believe

Velkee commented 1 year ago

Hi! From what I can gather from the thread Linux support is still very experimental and broken, but I want to report some issues I'm having while attempting to build the AUR package.

There seems to be several errors happening while trying to compile libzip for building premake5.

Error one appears as follows:

../../contrib/libzip/mkstemp.c:76:8: error: call to undeclared function 'getpid'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        pid = getpid();
              ^

Error two states this:

../../contrib/libzip/zip_close.c:621:2: error: call to undeclared function 'close'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        close(tfd);
        ^

If this is completely unrelated then please excuse me.

sebastien46 commented 1 year ago

Hi! From what I can gather from the thread Linux support is still very experimental and broken, but I want to report some issues I'm having while attempting to build the AUR package.

There seems to be several errors happening while trying to compile libzip for building premake5.

Error one appears as follows:

../../contrib/libzip/mkstemp.c:76:8: error: call to undeclared function 'getpid'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        pid = getpid();
              ^

Error two states this:

../../contrib/libzip/zip_close.c:621:2: error: call to undeclared function 'close'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
       close(tfd);
       ^

If this is completely unrelated then please excuse me.

When trying to build from AUR this error occurs for me too. From reading the log, I think this error occurs because premake-core uses a very old version of libzip from 2015 (0.11.2 to be precise), so it needs to be updated to get past this error (see #2121).

bwrsandman commented 12 months ago

AUR issue has been resolved. There's a needed patch for the POSIX console main to remove the use of the logger to fix a linker error. That should be looked into.