Syscall and HLE callback handler

[dogtopus]

Problem

For syscalls alone, #2 is a bit unclear of what it does, and after some elaboration, some parts of it don't make much sense and unnecessarily complicates things (like multilayer syscall handling and letting syscall handler's process() instead of using the callback object itself to handle host-guest data conversion).

We also wanted to have a unified way to handle guest callbacks for both syscall and HLE modules. This proposal takes both use cases into account.

Proposal

Some basic points

• Inside the Unicorn hooks, queue an event for the main loop to process and trigger a scheduler yield by calling uc.emu_stop(). Then in the main loop, run the appropriate GuestRequestHandler, schedule the next thread or panic depending on the yield reason.
• Both handlers have a list of available modules but behave differently. Syscall handler holds a Sequence of handler classes that gets instantiated sequentially on creation, while HLE modules holds a Mapping of handler classes that need to be queried and loaded manually elsewhere.
• HLEModule does not need to map the memory pages. This should be either handled by the Loader or the Process (when mapping onto reserved pages for magic exit, etc.). The Loader will also be responsible for initializing the HLE module and installing it using GuestRequestHandler#register_guest_module() on the HLEModuleHandler instance.

Workflow

Emulator context: on_intr(2) or on_page_fault() -> set flag and signal the emulator to stop Emulator context main loop:

stop due to svc ->
  extract syscall num and reformat stack frame ->
  self._states.grh_syscall.process_request(syscall_num) ->
  look up corresponding GuestModule object ->
  GuestModule#process(syscall_num) ->
  GuestCallback#respond_to(self._uc) ->
  register/execute coroutine from process_request with Scheduler

stop due to code hook set up around the allocated magic address (needs to at least have x permission set) ->
  self._states.grh_hle_func.process_request(addr) ->
  look up corresponding GuestModule object ->
  GuestModule#process(addr) ->
  GuestCallback#respond_to(self._uc) ->
  register/execute coroutine from process_request with Scheduler

Modeling

GuestRequestHandler:
A _uc
A _states
A _table: dict[int, GuestModule]
A _modules: list[GuestModule]
M __init__(_uc, _states)
M process_request(req_key) -> query _table and run GuestModule#process(req_key)
M register_guest_module(module: GuestModule) -> read available_keys, do intersection check and index them in table -> return index in _modules
M unregister_guest_module(module_index: int) -> fetch _modules[module_index] and delete all entries in _table before del itself

SyscallHandler(GuestRequestHandler):
A ENABLED_MODULES： list[SyscallModule]
M __init__(_uc, _states) -> super() -> self.register_guest_module(module) for module in ENABLED_MODULES

HLEFunctionHandler(GuestRequestHandler):
A AVAILABLE_MODULES: dict[str, HLEModule]
M find_guest_module(name) -> search AVAILABLE_MODULES

GuestModule(Protocol):
G available_keys: set[int]
M process(key) (usually this will be only one line, no need to subclass it tbh)

SyscallModule:
A _uc
A _states
A _table: dict[int, GuestCallback]
M __init__(_uc, _states) -> scan methods decorated with @syscalldef, make GuestCallback objects for each entry and store them in _table
M process(key) -> get from _table and passthrough to GuestCallback#respond_to(self._uc).
G available_keys

HLEFunctionModule:
A _uc
A _states
A _table: dict[int, GuestCallback]
A _names: dict[str, int]
M __init__(_uc, _states, base) -> scan methods decorated with @fcndef, make GuestCallback objects for each entry and store them in _table with an allocated address based on base. Extract names set by @fcndef and populate _names with the names and allocated addresses.
M process(key) -> get from _table and passthrough to GuestCallback#respond_to(self._uc).
G available_keys
M sizeof -> calculates the minimum memory size required for this handler. This will not be page-aligned.

GuestCallback:
A callback
A ret
A args
M __init__(callback, ret, args)
M respond_to(uc) -> coroutine. parse APCS args, pass result to underlying callback, await if needed, and set return value

[dogtopus]

There are also requests that need to be handled asynchronously (specifically any handlers that need to hand over the control to other threads while still be able to return results at a later time e.g. anything related to synchronization and things like Buzzer() which delays or yields in syscall via OSSleep())

We need to change this proposal slightly to handle this preferably with asyncio and perhaps a separate emulator execution thread controlled by an asyncio event loop or alternatively a dedicated emulator event loop driven by callbacks (instead of a regular loop currently being used) with bare Coroutines triggered before executing guest code in a tick(), which also has the ability to end a tick() when it blocks on a domain-specific Future object for Scheduler.