Towards a comprehensive debug suite

[RetroEdit]

One of the goals for mGBA is a comprehensive debug suite. I would value this feature greatly, and have some ideas.

I separate this into three lists: "required" and "nice to have", and "theoretical". Hopefully my classifications are sensible; I base them off of having spent time using the debuggers of other emulators, and then spending a fair bit of time attempting to use mGBA's debugger.

One may notice I also filed #1117, but this current issue is a result of significantly more experience using emulator debuggers in practice, and has a slightly different (and hopefully more focused) scope than the previous issue.

For me, a comprehensive debug suite requires:

• [ ] Step into, Step over, Step out functionalities with keyboard shortcuts. Many emulators use the function keys for this functionality, but those are already reserved for savestates here, so I'm not sure what the best default option (maybe replace savestate shortcuts, as debugger shortcuts are imo higher priority? At the very least, allow the user to rebind the shortcuts).
• [ ] Conditional breakpoints that are easy to use with good documentation.
• [x] Stack view, preferrably with indicators of when subroutines were entered (like Return from $xxxxxxxx labels on the return addresses, preferrably with proper handling of virtual method table jumps).

Some features that would be nice to have:

• [ ] The ability to cleanly dump portions of code/data of disassembly. Currently, data is technically possible through the hex editor, but it is not able to be directly used simply from a disassembly. Currently, code can be painstakingly extracted instruction-by-instruction (or piecemeal using trace).
• [ ] Easy modification and reversal of instructions, and a "move PC" function. Both technically possible currently, though inconvenient.
• [ ] Code-data logger (see also: #1629, #1714). BGB actually allows highlighting the UI for code and data separate colors, so something similar could potentially be an additional UI feature.

Some theoretical featueres I don't really see elsewhere:

• [ ] A method for detecting the beginning and the end of a function from mid-function (and potentially allow straightforward dumping of a function with the right command).
  • [ ] A heurestic approach: most GBA games I've seen start by storing various registers in stack, and then restoring those same registers (evaluating pairwise massively reduces the odds that data will be erroneously evaluated as such code). Some simple function will not push anything to the stack and instead end with a simple bx r14. In rare cases, these will fail, but it would work well enough in most cases.
  • [ ] Alternatively, and probably more straightforward, each long jump could be logged (or each active long jump, or the most recent long jump) to determine at least the beginning of functions definitively
• [ ] Usable symbol handling for virtual method tables. This is not a feature I've seen anywhere in particular, so it's highly theoretical. I see no theoretical barriers that make this impossible, though, and I have some vague notion of this in my head. I can imagine working on this myself if no one else has any interest (I can imagine working on any of these, although it may be a while before I make any contributions to mGBA).

This is by no means a comprehensive list of what I'd like to see it the debug suite. There are many potential features, but I do not want to make this post unreadable, and I feel like my effort in composing such a post would be better spent by attempting to implement such features myself.

[matheusmoreira]

mGBA already supports acting as a GDB server. Would be great if it could also integrate with modern reverse engineering frameworks like Radare2 and Ghidra.

Wouldn't this be more efficient compared to every emulator independently developing their own debugging tools?

[RetroEdit]

I will admit I am not sure how to use mGBA's GDB functionality. However, my impression is that the implementation is not ideal. Certainly, I will claim it's much easier to use a well-integrated debugging interface where standard functionalities are laid out in an intuitive way, though it might not be as powerful as GBD. It is also, as I understand, the ultimate goal for mGBA's debugging tools.

I welcome the idea of integrating static analysis tools. Something like that seems ambitious enough to be an entirely separate project from the short-term goals for mGBA's debugger; integrating tools like this well is not trivial. Additionally, when I spent some time using Ghidra, I found the lack of handling for virtual method tables to be a severe limitation (though I may have missed something, but my ideal setup didn't seem possible, making it a severe limitation; of course, I could eventually write my own if none exists, as I noted in the previous issue).

In any case, it's work someone has to do, so progress will largely be shaped by the development priorities of those actually working on mGBA.

[YoshiRulz]

The Debugger in EmuHawk uses external libdarm for disassembly. edit: So it would be nice to replace that with mGBA's own solution.

see also TASVideos/BizHawk#2312

[endrift]

Ok? I've already written a disassembler. And forcing people to use a third-party program with its own release schedule when I've already got progress towards it on my own debugger doesn't really sound like a solution.

[RetroEdit]

BizHawk's debugger is super janky and broken in my experience. I've never gotten it to work consistently or productively. mGBA seems to have a better debugging infrastructure than BizHawk's current stuff.

[matheusmoreira]

People are already using mgba with Ghidra in order to reverse engineer GBA games.

• A first look at Ghidra's Debugger - Game Boy Advance Edition

Looks like it already implements every requirement described in this issue. Not sure about the nice-to-have features though. It should be possible though.