EE Memory Base changed, breaks compatibility with tools and cheat tables

[Xeeynamo]

PCSX2 version: v1.7.0-dev-164

Description of the issue: Previous to #3523 merge, we were able to create tools that were able to attach to PCSX2 process to read and write in real time values from the EE RAM at 0x20000000. This was very convenient and hasn't been changed for at least 10 years from what I remember. Due to that change, basically all our tools and cheat tables will break. Most of our tools (and more) depends from this specific address.

Proposed solutions:

1. Keep the content of that pull request, but make (uptr)MainMemory()->GetBase() to return 0x20000000 (this is from SysMainMemory contructor, in System.cpp).

2. If the intention is to not fix it and stick with a more dynamic address solution, provide an API to return a specific mapped memory location. This can be provided as an exported function or via TCP from a specified default port.

Last known version to work: Before #3523

[lightningterror]

See https://forums.pcsx2.net/Thread-Is-memory-handled-differently-in-the-1-7-0-builds?pid=612571#pid612571

[Xeeynamo]

See https://forums.pcsx2.net/Thread-Is-memory-handled-differently-in-the-1-7-0-builds?pid=612571#pid612571

I am afraid that the proposed solution to implement an IPC would costs a consistent rewrite of all our existing tools, it would break all the closed-source and currently un-maintained tools and breaks for ever any Cheat Table.

[lightningterror]

Anyway the changes were done to get the recompilers working on 64bit builds, a bit unfortunate that it breaks compatibility with many modding tools.

[tadanokojin]

tagging @tellowkrinkle

[Xeeynamo]

Anyway the changes were done to get the recompilers working on 64bit builds, a bit unfortunate that it breaks compatibility with many modding tools.

If this will not be fixed to map it back to 0x20000000, can we at least consider the second option I proposed? The solution that @tellowkrinkle proposed to the forum, which suggests to attach a debugger and read from HostMemoryMap::EEmem, might require elevated permissions, a debugger and would not work easily (or at all) on modern managed languages .

[tadanokojin]

I didn't really look at it but what about #3591?

Edit: apologies missed the link

[GovanifY]

@Xeeynamo

I am afraid that the proposed solution to implement an IPC would costs a consistent rewrite of all our existing tools

Except it probably won't, you can Appinit_DLL on windows ReadMemory and whatever, since your tools were windows only, and have pcsx2 ReadMemory to IPC translators under the hood in libraries, which will make you able to run even closed source programs this way.
Now I believe you understand why I talked so adamantly about it and said that was a bad idea in the past :)

[Vladabdf]

@Xeeynamo

I am afraid that the proposed solution to implement an IPC would costs a consistent rewrite of all our existing tools

Except it probably won't, you can Appinit_DLL on windows ReadMemory and whatever, since your tools were windows only, and have pcsx2 ReadMemory to IPC translators under the hood in libraries, which will make you able to run even closed source programs this way. Now I believe you understand why I talked so adamantly about it and said that was a bad idea in the past :)

Our tools are not Windows only. Furthermore, +1 to the idea of either reverting this change or providing an API to work around the problem. This change may not be super devastating for making simple pnach codes but it will break compatibility with tons of other things.

[GovanifY]

@Vladabdf

Our tools are not Windows only.

I'd like to know where exactly out of the WinApi you find WriteProcessMemory.

EDIT: For the OpenKH people that wonder, here are windows only snippets: https://github.com/Xeeynamo/OpenKh/blob/master/OpenKh.Tools.Common/ProcessStream.cs https://github.com/Xeeynamo/KH3SaveEditor/blob/master/Tooling.MemoryWatcher/ProcessStream.cs https://github.com/Xeeynamo/OpenKh/blob/a8de1e5cbc847b7ee87a16e29182f806d5b34e07/OpenKh.Tools.DpdViewer/OpenKh.Tools.DpdViewer.csproj#L6 ^ and all other GUIs that use WPF in the OpenKH project, being all of them.

[Delta-473]

If the emulator just exposes the base address then existing tools can just get it from the emulator and do there thing again in a way that isn't so fragile to break by just moving around the base address. For old tools that have been abandoned by there author but are still in use, a simple wrapper program can maybe solve the issue there. But this looks like the best solution for all parties, that isn't to much work and is cross platform.

[GovanifY]

@Delta-473 Exposing it would have to be done in some way that requires IPC, so it'll probably use my socket IPC. For new tools there isn't a big reason not to use this protocol over the OS internals process memory tools, which would also be safer. As I already mentioned above in this issue I can make a library wrapper for old usages of the libraries for pcsx2 < 1.7.0, which replaces the *processmemory by calls to the ipc, transparently. Windows would be AppInit_DLL and posix LD_PRELOAD, which would make everything work transparently without needing to edit any source code of any kind.

This is one of the main reasons why the IPC was discussed and implemented in the first place, I fail to see why you're reluctant to its usage, unless this is due entirely to personal grudges.

[Delta-473]

No it was just a question. I trying to understand everything correctly. thx for you time.

[Xeeynamo]

@Vladabdf

Our tools are not Windows only.

I'd like to know where exactly out of the WinApi you find WriteProcessMemory.

EDIT: For the OpenKH people that wonder, here are windows only snippets: https://github.com/Xeeynamo/OpenKh/blob/master/OpenKh.Tools.Common/ProcessStream.cs https://github.com/Xeeynamo/KH3SaveEditor/blob/master/Tooling.MemoryWatcher/ProcessStream.cs https://github.com/Xeeynamo/OpenKh/blob/a8de1e5cbc847b7ee87a16e29182f806d5b34e07/OpenKh.Tools.DpdViewer/OpenKh.Tools.DpdViewer.csproj#L6 ^ and all other GUIs that use WPF in the OpenKH project, being all of them.

May I ask how links to repositories of my account is related to the issue? I just asked to expose an API or give back the previous address rather than a new one.

[GovanifY]

@Xeeynamo This is related by the requirements of the library wrapper needed. I am showing that you only need AppInit_DLL library wrappers, since windows support has always been the only scope of any projects you linked. If you find any linux implementation please do tell and I'll implement ptrace library wrappers as needed.

Now that this issue has been marked as won't fix and is resolved, I don't see a point in keeping it open.

[ghost]

I do believe this is a bad idea to go around. As a change in memory address by making it a dynamic one breaks all of the previous tools of modding (Such as EmuHaste) and already established cheat tables (Such as those for REing Soulcalibur games) in an unfixable way. I am +1'ing the idea of either mapping the memory back to 0x20000000, or at least make the memory a static one so tools and cheat tables can be fixed accordingly [Recommended]. Or if that is inconvenient, allow for an API call which returns the current game memory location [Not recommended because CE and EmuHaste].

[TellowKrinkle]

While you could map memory back to 0x20000000 on 32-bit PCSX2, you will not be able to so on 64-bit PCSX2 as the recompiler requires its memory to be within ±2GB of everything else (including C++ global variables that it needs to access).

And I realized that the console printout of the base address is only enabled in debug builds, I'm happy to make it print in all builds if that would help.

[Xeeynamo]

While you could map memory back to 0x20000000 on 32-bit PCSX2, you will not be able to so on 64-bit PCSX2 as the recompiler requires its memory to be within ±2GB of everything else (including C++ global variables that it needs to access).

And I realized that the console printout of the base address is only enabled in debug builds, I'm happy to make it print in all builds if that would help.

Printing the memory address to stdout would only work at start-up and not if the application is already running. Plus it would only work if programmatically read (eg. not on Cheat Engine).

My bet is that every one would just be happy if we still have 0x20000000 on 32-bit builds and another pre-defined fixed memory address for the 64-bit builds. I am going to spend a long week-end outside and I will not have enough time to fork and propose a change, but I will next week 🙂

[TellowKrinkle]

64-bit will not be able to have a pre-defined fixed address unless you can convince the various system linkers to put PCSX2 itself at the same address across all OSes

[legend800]

This also broke compat. with the Mouse Tool which allows you use the mouse to control the joystick, which is very nice for rail shooters like Time Crisis - Crisis Zone. Other "joytokey" type programs don't allow 1:1 in-game cursor movement (absolute/relative) like this app.

http://forum.arcadecontrols.com/index.php/topic,149714.msg1654496.html#msg1654496

Compounded by the alternative (nuvee guncon plugin) being unavailable now after usb plugin support was dropped.

[GovanifY]

An alternative is currently in development with the use of IPC. It also aims to be cross compatible with RPCS3 once features land in master

[F0bes]

Closing this and the related pr, read here.

[legend800]

So they'll be no alternative or way forward on this now that the app is 64bit only?

[F0bes]

Currently no. We only have IPC. I haven't looked into it at all, but I was wondering if we could use this method to export the base address? I'm unsure if that value needs to be constant or not, again, didn't look into it.

[TellowKrinkle]

Should work fine if Cheat Engine can read those Globals are here, just need to stick a extern "C" _declspec(dllexport) on them I guess https://github.com/PCSX2/pcsx2/blob/1e198fbb839ad5bd97592fbb1eafcb6243f67a6b/pcsx2/System.cpp#L334

[Delta-473]

Doesn't rpcs3 have a static base address? If something from there way can be used as inspiration, could be a solution. But I'm not familiar with neither code base.

[TellowKrinkle]

It may. If someone wants to rewrite the recompiler to support arbitrary positions for main memory, they're welcome to do so. It seems like a lot of work just for this IMO.