dirkwhoffmann / virtualc64

VirtualC64 is a cycle-accurate C64 emulator for macOS
https://dirkwhoffmann.github.io/virtualc64
Other
355 stars 33 forks source link

Emulator can crash in `finishInstruction()` #808

Closed dirkwhoffmann closed 1 month ago

dirkwhoffmann commented 2 months ago

Background: This is the beginning of the main execution function:

C64::execute()
{
    auto lastCycle = vic.getCyclesPerLine();

    try {

        do {

            // Run the emulator for the (rest of the) current scanline
            for (; rasterCycle <= lastCycle; rasterCycle++) {

                // Execute one cycle
                executeCycle<enable8, enable9>();

                // Process all pending flags
                if (flags) processFlags();
            }

            // Finish the scanline
            endScanline();

        } while (scanline != 0);

        // Finish the current instruction
        finishInstruction<enable8, enable9>();
        ...

In the last displayed line, finishInstruction is called. If the CPU is in the middle of an instruction, it runs the emulator for some additional cycles until the fetch phase is reached (again). This works under normal circumstances because VICII is at the beginning of a new rasterline when the function is called. The function looks like this:

template <bool enable8, bool enable9> void 
C64::finishInstruction()
{
    while (!cpu.inFetchPhase()) {

        executeCycle<enable8,enable9>();
        rasterCycle++;
    }
}

Now, assume the CPU's RDY line is pulled low (which freezes the CPU). On a vanilla C64, this is not an issue as VICII only does this during bad lines. However, cartridges can also pull down the line low. I.e., when a REU is attached, this happens for a long time. In this case, finishInstruction executes instructions until rasterCycle runs over, making the emulator crash with an out-of-bounds access to the VICII function table.

TODO: Evaluate if we can go without finishing instructions.