s18: mwalk hangs up

[jotego]

Soon after booting up, mwalk freezes during the demo when the dance scene begins. If a game is started, it will also freeze at some point although it is less predictable. Once frozen, the core only reacts to a reset. The test button (F2) stops working too.

[gyurco]

When the VDP control port is written with a 32-bit write, the CPU splits it to two 16 bit writes. The VDP sometimes hangs at the second write (dtack is never pulled). Probably it happens too fast after the first? It occurs less frequently if the clock recovery is turned off.

[jotego]

Nice catch! As VDP's clock is 96MHz and CPU's is 48, it won't be able to see some 1-clock pulses.

Do you remember that I had a pulse enlarger for VDP's DTACK signal in jts18_vdp before? When you worked in the module, you removed it. I guess you could try adding it back.

[gyurco]

DTACK is not a one cycle pulse, it's asserted until AS is released. But it's not asserted at all on the second VDP write. Unfortunately what's happening inside the VDP is a bit of black box, it can be a timing issue, or it's really a too fast re-selection. Strange why it doesn't happen in Alien Storm.

[gyurco]

I don't understand the cpu cen generator, most of the time it runs at clk/2 (24 MHz):

[gyurco]

Seems only when rom_cs is active, and it waits anyway for the ROM data...but still strange.

[gyurco]

I think it's because the control signals are probably glitching. I registered them in the VDP clock domain, works so far for some minutes.

[jotego]

I kept a long simulation running, keeping the signals around the time I expected the issue to happen. I have shared the waveforms and frame images in our Google Drive shared folder. You may run out of memory quickly if you plot too many waveforms. I will upload a shorter fst file this evening as I am re-running it with a shorter dump frame-span to save memory.

I can see the double VDP access that you described and indeed the VDP fails to produce the second DTACK signal. Before you made the changes to the VDP block, this was very common and happened at the first few frames after booting up. Maybe the issue just became less probable after your initial edits but it is still there.

Note that there are no glitches in this simulation as there are no delays due to combinational logic. Still, the VDP fails to produce DTACK and the system comes to a halt.

[jotego]

I don't understand the cpu cen generator, most of the time it runs at clk/2 (24 MHz):

When the CPU is waiting for SDRAM data, cpu_cen signals may run at full speed. It does not impact the CPU speed though because the CPU is halted by the high ASn signal. The average frequency fave (notice the BCD format) should still indicate 10MHz.

I think this could be modified to keep a normal cpu_cen rate during this time and only run the speed up after ASn comes down. I do not think it matters, though. You can try disabling RECOVERY, which will keep cpu_cen uniform no matter what (and effectively reduce the CPU speed).

As this core only targets the platforms with large FPGAs, the 64kB RAM memory could be moved to BRAM to reduce the cpu_cen throttling. The worst case for throttling is not ROM access by quick memory write bursts from the M68K, which can dump all registers to memory with some instructions. That generates a lot of time debt due to SDRAM latency. This is probably not related to this issue, though.

[gyurco]

I kept a long simulation running, keeping the signals around the time I expected the issue to happen. I have shared the waveforms and frame images in our Google Drive shared folder. You may run out of memory quickly if you plot too many waveforms. I will upload a shorter fst file this evening as I am re-running it with a shorter dump frame-span to save memory.

Interesting that it breaks even in the sim. Does the dump have all the internal VDP wires? Then with some CSI work, it could be pinpointed what went wrong.

[jotego]

Interesting that it breaks even in the sim. Does the dump have all the internal VDP wires? Then with some CSI work, it could be pinpointed what went wrong.

Yes, internal VDP signals are kept in that file. I am running another sim with your change too. I'll upload the fst file this evening. You should be able to compare the signals more easily that way.

The commit 5f5b6b2 did not fix the other games that do not boot, which could be a related issue. Although the legal message in ddcrewj2 is now visible. I do not remember seeing it before. It might be worth it to bring the AS/DTACK/vdp_cs signals to the debug_viewer.

[gyurco]

I left moonwalker running for ~30 minutes, and still not freezing. The other games actually don't hang, the CPU runs mostly in a tight loop.

[jotego]

Interestingly, the fix still gets to a halt in simulation. I copied the files to the shared folder with the git hash attached to the folder name. This is probably working by a fluke on the FPGA :-(

[gyurco]

Am I watching the same fst?

I don't see it's locked, the latest address is not even the VDP's.

[jotego]

You are right. I got confused by the frame images stopping at around that point. But of course, I had re-run the short sim I had just used for another purpose, which stopped at that point. That's good news. The fix may be working! I will re-run the sim today (the real long sim) to double check it.

[jotego]

Actually, the fact that it didn't halt makes that fst file perfect for comparison with the other one as you can see what went different inside the VDP and understand why registering those signals make a difference.

[jotego]

A long simulation over 10k frames did not crash. Registering the signals works both on FPGA and simulation.

[gyurco]

Good! Unfortunately the test.fst doesn't seem to have the full set of internal VDP signals (from mwalk-1bb5cb09), so I cannot pinpoint what went wrong.

[jotego]

We do not understand this but unless we find another lock-up instance in this or another S18 title, we can settle it here. @jtmiki mark it as done.