[N64] Possible buffer overflow in RSP DMA code - potential security concern?

[bkimmett]

Recently, a buffer overflow was discovered in another N64 emulator, ParaLLeL Launcher's, RSP DMA code:

https://gitlab.com/parallel-launcher/parallel-n64/-/commit/fa9e2e08b0c6cde117cb05cc071cdb74a9e90ad2

Prior to being fixed, this bug was successfully exploited to escape the emulator and perform ACE on the user's computer (see https://www.twitch.tv/videos/2147408942?t=12257s for an example of this escape in action).

I'm not sure if the RSP DMA code in ares is subject to this buffer overflow or not, but I figured I should make you aware of it in case it is.

[rasky]

Thanks. Ares is not subject to this bug. There is a couple of reasons why Ares is more defensive than old emulators (or new emulators with old cores):

• Ares passes most low level test ROMs which excercise many corner cases, so most overflow conditions are covered there and implemented correctly (btw, the fix you linked does prevent the bug, but is wrong because RDRAM access is not masked that way on real hardware).
• Ares hardware registers and memory accesses always go through some abstractions that make sure that overflows are correctly handled (for instance, the RSP DMA memory register uses our n12 special data type which is a 12-bit integer, so it can't ever overflow 12 bits). This makes the code easier to write correctly by default.

We can't rule out the existence similar bugs of course, but I would say in general we are already pretty defensive.