search

n64dev / cen64

Cycle-Accurate Nintendo 64 Emulator
BSD 3-Clause "New" or "Revised" License
803 stars 70 forks source link

Rewrite IS-Viewer 64 console output emulation #232

Open parasyte opened 2 years ago

parasyte commented 2 years ago
rasky commented 2 years ago

The unfortunate reality of the N64 homebrew scene is that in the past two years, we have built many different tools around the (wrong) implementation of ISViewer. Not only the same (wrong) implementation is available in basically all major emulators, but it was also adopted by libdragon and by the Modern SDK (modernized libultra) and basically all ROMs made in the past two years are linked with a library that can produce output in the (wrong) ISViewer format. I personally spent lots of effort to standardize ISViewer across the whole ecosystem to make N64 homebrew development less miserable.

The problem with fixing the current implementation to adhere to the real hardware is that we are breaking every ROM that was made in the past two years. And even if we decide that we can live with that, we are creating a schism: everything made before the fix will not work with everything made after. And everything made after, will not work with emulators made before. This means that for the next year or two, all homebrew developers will face troubles: they will update their emulators and their development tools will stop to produce output; they will be forced to upgrade their tools even if they didn't want to, or downgrade the emulators. Then they will try to compare emulators, but new versions of other emulators maybe they would still support the older protocol. People will download repositories from GitHub, compile them, and will not get output in emulators because that repo was made before the schism; to get output, they would be force to update the repo to the latest development tool, which might be hard / impossible if APIs changed, so they will have to download old versions of emulators.

Changing ISViewer now will create trouble in the homebrew community for the next two years. But still, I think it could still be acceptable if there was a greater good to pursue. In this case, I fail to see what is the benefit. A theoretical benefit would be to be accurate with respect how an obscure development tool used 25 years ago, basically impossible to buy today, and basically impossible to use even for the few people that own one. Honestly, I do not see great value in this: it seem a really purist approach that fails to acknowledge the damaging ripple effect that it would create.

I am all for accuracy and LLE emulation, but I am also a practical person. The N64 homebrew is blossoming now, and it is great to see people that in a few minutes can accomplish what it took days just a few years ago. Let's not regress for a dubious improvement. Let's simply rename the current ISViewer implementation with another name, let's call it HomebrewStdout or whatever, and drop all temptations of aligning it to a dead hardware unit that nobody knows of and nobody will ever use again.

I'm happy to provide a patch to OoT to align it with the current HomebrewStdout protocol so that all hacks will be able to see their output in all current and future emulators.

parasyte commented 2 years ago

The problem with fixing the current implementation to adhere to the real hardware is that we are breaking every ROM that was made in the past two years.

I'm going to be very real, and this is probably something most people will not want to hear but it has to be said. I don't care about bad hacks and emulator-only homebrew. Full stop.

Here's a short story to illustrate why I hold this position. When I was cutting my teeth about 25 years ago, the best emulator and general-purpose hacking tool we had for NES was NESticle. Unfortunately, NESticle is notoriously inaccurate. But its pattern table editor made it an obvious choice for editing old games. This lead directly to the problem that many hacks and homebrew developed with NESticle were then dependent upon NESticle. They do not run on a real NES, nor emulators that don't behave like NESticle.

Over time, the community wrote better emulators, and created better tools. It's now possible to produce high quality NES homebrew using nothing more than a modern emulator and you can more or less guarantee it will be fully compatible with the NES that has been in your closet since 1993.

What's more is that no one remembers any of the broken NESticle-only homebrew and ROM hacks. In the long term big-picture, NESticle was a short blip that came and went. The same will be true of incorrect IS-Viewer 64 implementations as emulators continue to adopt more accurate emulation of the hardware.

It is (and always has been) more important to preserve the hardware than any individual game or piece of software. You have a chance to course correct now at the minor cost of confusion for the next 2 years. In 25 years, no one will remember or care, but they will be able to understand how the IS-Viewer 64 actually worked just by reading through the code.

I'm happy to provide a patch to OoT to align it with the current HomebrewStdout protocol so that all hacks will be able to see their output in all current and future emulators.

Having a "homebrew-only" console protocol seems like a good idea. As long as it isn't something too crazy that cannot be replicated in hardware if someone chooses to implement it on a cartridge. But IS-Viewer 64 isn't that. The existing protocol is more complex for a hardware implementation than it needs to be. For instance, the atomic memory clear when updating the write head requires double buffering. And the existing protocol also directly conflicts with IS-Viewer 64, so there is that.

rasky commented 2 years ago

To bring the discussion forward, I've tried to come up with a suggestion for this specific PR:

I am absolutely not happy with this suggestion, but it is a compromise. It will create an emulator with two different conflicting implementation for a debugging protocol, and I don't think anybody will understand why we need two similar but conflicting emulator-only debugging protocols in the N64 scene, just because one of the two is possibly more similar to a dead peripheral that nobody owns or operates. But at least this solution doesn't break what we built, and just builds upon it.

snacchus commented 2 years ago

I generally agree with @rasky that just changing the ISViewer implementation to be more hardware accurate would currently have harmful consequences to little benefit. We could of course just say "If you're doing homebrew development, use another emulator that still implements the old protocol", but the truth is that cen64 is currently an invaluable tool for homebrew because of its gdb support.

On the other hand I also agree that hardware preservation is important, so I don't want to just say that this PR shouldn't be merged. But it would be foolish to ignore the consequences this would have, even if just short-term. In an ideal world, emulators would implement both the original, hardware-accurate ISViewer protocol for hardware preservation purposes, and a separate homebrew protocol. Keeping the latter separate would offer the benefit of extensibility. It seems obvious to choose what is currently (incorrectly) called "ISViewer" as this homebrew protocol, since it is already implemented by many emulators and supported by homebrew tools. The compromise that Rasky suggested would be a good start to reach this ideal situation. But then we must be absolutely careful to document this change in a way that minimises confusion among homebrew devs. For example the help text for the flag that activates the hardware-accurate protocol should make it clear that it shouldn't be used for homebrew purposes since it is not supported by other tools.

parasyte commented 2 years ago

I need some more time to digest why this is so controversial. Regardless I will address your points, it's the least I can do.

  • This has absolutely nothing to do with homebrew ROMs that don't run on real hardware. I have been working hard in the past years to help emulators align to real hardware like you described in the NES scene, it is one of my main goals. I have zero interest in emulator-only development, and I don't understand why you feel it is related (while still agreeing that a hombrew-only protocol -- which by definition is emulator-only -- is a good idea).

I made a comparison to NESticle for illustration purposes. There was never a point when I intended to imply this has anything to do with homebrew that doesn't run on console hardware. We agree that HomebrewStdout is emulator-only, and that's the full scope of it. No need to go any further on that subject.

  • There is no need to replace HomebrewStdout with ISViewer for homebrew development. Trying to do that would create the schism I described and it would be detrimental, for absolutely no benefit (there is a severe scarcity of real ISViewer, and they are also basically impossible to operate because of the dead SCSI interface).

I have a big problem with this point, and it has taken quite a while for me to figure out how to respond to it respectfully. You're viewing this PR from the perspective of a homebrew developer. While there is nothing wrong with that view per se, it is disingenuous to suggest the purpose of this PR is to replace HomebrewStdout for homebrew developers. They happen to be caught in the crossfire due to the design of HomebrewStdout, they are not the target audience.

Secondly, scarcity and SCSI are beside the point. The protocol is software-driven on both sides and is entirely independent of the physical layer. One could adapt the IS-Viewer 64 protocol to any development cartridge with just 64KB of memory and a microcontroller with their choice of physical connection. It would most likely it be RS-232/USB or even WiFi. It could be SCSI if they had some wild reason for it, but that's not a requirement in any way.

This is by far your weakest argument.

  • It would be great to enhance HomebrewStdout with more features (and I agree with your suggestion that it would be great if they still were designed in a way that make it feasible to implement them in hardware, if one wishes so). Planning so should not be bound to the hardware features that were available in ISViewer in the '90s.

Yes, but to reiterate, homebrew developers are not the target audience here. Discussions on how to improve HomebrewStdout should probably not derail this one. It would be helpful for more productive conversation if you could shift your frame of reference away from solely what a homebrew developer sees here.

  • I have personally zero need to have an accurate emulation of ISViewer, but obviously others like you may and you obviously must be free to add that to emulators if you wish so, even though this will create endless confusion.

I have personally zero need to have HomebrewStdout that uses the IS-Viewer 64 address space. It is these kinds of comments that grate on me.

Your proposal sounds fair. I think an adequate solution would be just dropping all references to "IS-Viewer 64" in the emulator, documentation and tutorials, in libdragon, and in any homebrew that uses the terminology. Keep the implementation as-is, maybe warn that it conflicts with (but is not compatible with) IS-Viewer 64 if you want. But a rename would prevent how I was mislead while researching IS-Viewer 64 hardware. I used the cen64 implementation as a source of truth for that research which lead directly to this PR.

One final bit of feedback, @rasky please try to be less antagonistic with contributors. I am personally considering not contributing again if this kind of work is going to be called dubious and create a mountain out of a molehill. I'm also trying to understand how you are speaking on behalf of all homebrew developers in this situation, or why it even matters. There are other stakeholders who have opinions that are just as valid as your own.

tj90241 commented 2 years ago

Just my 2 cents -- even though I'm not really active with the scene or project anymore... the goals of the project were always accuracy first with a focus on hardware preservation.

As much as it may hurt to do the right thing here, I'm fully of the opinion that this should merge. ROMs should be recompiled if they do not work anymore.

loganmc10 commented 2 years ago

We had a similar argument with mupen64plus when I proposed a change to enforce PI DMA alignment - https://github.com/mupen64plus/mupen64plus-core/pull/541 - this fixed an issue with Taz Express, but broke quite a few ROM hacks.

My position was always that those ROM hacks were "Project64 games", not "N64 games". We weren't making a Project64 emulator. Especially for an accuracy-focused emulator like CEN64, it makes no sense to maintain behaviour that doesn't match real hardware.

You don't want to make an emulator that plays CEN64 games, you want an emulator that plays N64 games.

rasky commented 2 years ago

You're viewing this PR from the perspective of a homebrew developer. While there is nothing wrong with that view per se, it is disingenuous to suggest the purpose of this PR is to replace HomebrewStdout for homebrew developers. They happen to be caught in the crossfire due to the design of HomebrewStdout, they are not the target audience.

So, who is the target audience? I am not sure I have fully understood the purpose of this PR. To the best of my understanding, this PR has the goal of doing accurate hardware preservation of the physical ISViewer64 peripheral that used to exist in the 90s. Please confirm if this is the case, because I do not want to have further arguments in the future about this code. If we agree the goal is hardware preservation, please let's also state this at the top of the file, so that we will all agree on further changes that will improve accuracy with respect to how ISViewer64 operates, even though they step on future hardware reimplementations and whatnot.

Your proposal sounds fair. I think an adequate solution would be just dropping all references to "IS-Viewer 64" in the emulator, documentation and tutorials, in libdragon, and in any homebrew that uses the terminology. Keep the implementation as-is, maybe warn that it conflicts with (but is not compatible with) IS-Viewer 64 if you want. But a rename would prevent how I was mislead while researching IS-Viewer 64 hardware. I used the cen64 implementation as a source of truth for that research which lead directly to this PR.

Agreed on all points. If you agree, I will send a PR first to rename the existing implementation in cen64, and then you can rebase yours on top.

cuavas commented 1 year ago

Primarily the read head is updated after bytes are consumed from the ring bugger.

Sorry, I know it's immature, but I laughed.

KajeArch commented 1 year ago

So, what happened to this PR @parasyte @rasky ? It's clear that this PR and proposal needs to be accepted. Cen64 stands for accuracy, doesn't it?

ghost commented 1 year ago

So, what happened to this PR @parasyte @rasky ? It's clear that this PR and proposal needs to be accepted. Cen64 stands for accuracy, doesn't it?

NO