Linux Client need modification

[tioui]

The Archipelago Client use the Windows only Pymem library to read location check. If we want a Linux (or other Posix system) port, we will need to replace it with another thing. Here three scenarios I propose:

• Find a multiplatform (cross os) library that works the same on Windows and on Linux.
  • I search a little bit on the internet and the only thing that I find is this: https://github.com/qb-0/PymemLinux but it has to be launched as root. I think that we should make this a last resort option.
• Change a little bit the client so that it starts the game (gk executable) himself and use Python-ptrace in Linux to read the memory instead of Pymem.
  • The problem with that option is that we will need to do the same code twice: on with Pymem for Windows and another with Python-ptrace for Linux.
• At last, changing the client structure to use only the repl connection (using the Web API) to read and write in the game variable. That way, The Pymem is not needed anymore to read in the game process memory.

[massimilianodelliubaldini]

• [ ] Use subprocess calls to ps to find the gk and goalc processes to ensure they don't die on us (rather than pymem)
• [ ] Use the REPL to query the *ap-jak1-info* memory structure and parse for your progress.
  • [ ] Prerequisite: Would need to modify the the REPL server itself to respond to requests properly, and with actual data not just "OK!"

[massimilianodelliubaldini]

It might actually be easier to just put in platform-specific python modules - Pymem for Windows, ptrace for Linux. I'm not against it as an option.

[massimilianodelliubaldini]

Though this kind of scares me:

This project is no longer maintained and is looking for a new maintainer.

[tioui]

It had an update last march. https://github.com/vstinner/python-ptrace/commits/main/

[tioui]

Also, we could probably use https://proc.readthedocs.io/en/latest/api.html#module-proc to know if a process existed instead of the subcommand hack.

[Minimistro32]

I can see there is help wanted here. I'm curious, approximately how much code is using Pymem and would need to get a Linux version using Pytrace? I don't have a lot of spare time, but I do have python and some low-level programming experience.

From what I can tell at a quick glance: MemoryReader.py, ReplClient.py, and Client.py are the affected files. That sound right? No promises yet, just trying to scope out what size a project linux support might be.

[tioui]

I did look at the beginning of winter, but massimilianodelliubaldini was actually working hard on it and was doing a lot of modification really fast. So, I decided to pause a little on this project to work on my other project. I think that the code is quite stable at the moment, so it could be a good time to think about it again.

The last time I looked at it, there was modifications to do in almost every files of the client. A lot of hard-coded ".exe" and the use of power shell that I think is not necessary under Linux.

One of the problems is that the processes are a lot more secure from other processes (of what I know) and if on Windows, you can just start looking at process memory, it is not possible in Linux if you don't have root access or if the process looking is not the parent process of the process that is looked. So, this is a challenge too, it sill is needed to change the starting process.

Another problem is to be sure that the memory structure of the game process is the same as when compiled on Windows. If it is the case, it will probably be easier to create a routine that manages the memory reader for Windows and another one for Linux. If the process memory is not the same between Windows and Linux, we will have to change every place in the code that is reading some code or more likely since a structure is used in the game code, using relative addressing with a different base address.

In other words, lots of think to change and test.

[Minimistro32]

I see that is a lot to consider. Respectfully might be more than I can handle right now, especially if the memory structure is actually different like you mentioned.

That said, have you seen PyMemoryEditor? I saw it mentioned in this Pymem discussion post. At a glance, it has Linux and Windows support for reading, writing, and searching memory. I'm nowhere familiar enough with this project to say. Do you think it could be robust enough to be used here?

I will say, PyMemoryEditor has the same caveat as Python-ptrace: it needs to be the parent process in order to function. But it has the upsides of being cross platform, actively maintained (near as I can tell), and would result in one codebase which can be a plus. It'd effectively be combining your first two ideas from this post, which seems to me a better alternative than fixing the REPL like Markus noted.

[tioui]

I did not know PyMemoryEditor. It is absolutely something we should be looking into. I think that using multi-platform tools are always better than using multiple separate tools for every platform. But since that impact the Windows build as well as the future Linux one, @massimilianodelliubaldini should accept that we change the Pymem usage. I would not like to work on something that would not be accepted in the repo.

As for the alternative of fixing REPL, I think that the best thing that could be done is actually to fix the REPL server to accept queries as well as commands. This would be better for REPL and better for the AP GOAL clients. The problem is, I looked at it and we are talking about massive modifications since the entire query structure are done with prints.