Plans to finish full NSF playback implementation?

[sofakng]

I was extremely excited to see this project because it seems to be the closest implementation of a hardware-based NES music player but it seems like the NSF playback function isn't 100% complete?

Are there plans to finish the implementation?

I've heard that VGM files don't have the 100% information for completely accurate playback but NSF does.

[dkonigsberg]

VGM and NSF are different styles of format. VGM is basically a "recording" of the commands being sent to the audio synth hardware, whereas NSF is actually a rip of the code that generates those commands. I'm not sure what limitations of VGM you've heard of specifically.

The real issue with the NSF support right now has to do with the playback of sample data (APU DMC) that some games use. It needs to be loaded in advance of being played, obviously, and the NES CPU doesn't have enough RAM to pre-load all the data that could theoretically be there.

With VGM, I can pre-analyze use of all the audio sample data in the track and decide when/how to load it into the NES CPU's memory space (0ab51e1f31eb3a67d4754d60677abc206d0d6d5c). This was quite complicated to do, but it works.

With NSF, there's really no easy way to do this because NSF is more of a "program" and less of a "data format". The only way to realistically pull it off would be to either re-engineer the Nestronic to give the NES CPU a lot more RAM and simulated mapper hardware, or to emulate a 6502 on the ESP32 side significantly faster than playback speed so I could sneak data loading within the playback process.

[sofakng]

Thank you so much for the detailed response!

I've spoken to Jose Tejada who is an FPGA developer (MiSTer project, etc) and has created FPGA implementations of many sound chips including YM2203, YM2612, YM2610 and YM2151. I asked him about developing an FPGA-based VGM player and this was his response:

I am afraid that these formats are actually rather inaccurate. They were designed with a fixed sample rate of data commands at 44kHz. This adds jitter to the sound phase, I am not sure how noticeable it is. Almost no actual system was sending data at 44kHz. Double Dragon, for instance, sends it at some submultiple of 3.57MHz. It is even worse for some PCM encoding systems such as Megadrive's because VGM effectively resamples the data and creates some noise artifacts.

I didn't ask about the NES specifically so I'm not sure if this applies or not. However, I'd love to hear your feedback or information.

Regardless, I think this is an absolutely fantastic project and I really appreciate how much work you've devoted to this!

[dkonigsberg]

There is probably some truth to that gripe. However, I'm going to put this into the category of "problems that likely don't matter" because you're not directly playing audio samples at that rate. You're just sending commands to audio synth hardware at a much coarser rate, and its doing the business. In the context of the NES and VGM playback, 44.1kHz is really just the time base that is used to calculate delays between commands (when specified in the VGM data).

Besides, the whole architecture of the Nestronic is really not capable of doing direct NSF (or clock-cycle-perfect) playback anyways. Its designed around determining that directly sending APU commands to the NES CPU at a slower rate was good enough to produce audio that (to a human ear) was close to perfect. I really designed it around VGM, so the NSF player is actually running a 6502 emulator on the ESP32 side and essentially generating VGM-style data on the fly.

FWIW, I've played audio tracks from the modern chiptune game "Shovel Knight", which supplies an NSF version of the soundtrack, and it can keep up just fine. This game's soundtrack generates audio commands at a much higher data rate than any real NES game, so it makes for a good stress test.

In order to make something that could do cycle-accurate NSF playback, I'd have to completely re-engineer the device so that the NES CPU could run the NSF code directly. (Probably with a significantly larger SRAM on the NES CPU, and an FPGA in-between, rather than simply having the NES CPU run a program that communicates with the ESP32 to receive audio commands.)

[sofakng]

I'm a big fan of Shovel Knight and had no idea it supplies NSF soundtracks but that is very interesting.

You're answered my question(s) so we can close this comment, but I did have a related (somewhat off-topic) question:

Does Nestronic (ie. VGM format) support games that used mapper hardware? (ie. VRC6, etc)

You briefly mentioned this above, but if I understand correctly NSF playback would require mapper hardware (or emulated) to process those 6502 instructions, but VGM playback already has them "processed" and is the recorded output from the mappers, etc?

[dkonigsberg]

Does Nestronic (ie. VGM format) support games that used mapper hardware? (ie. VRC6, etc)

So the mappers do two things:

1. Provide a way for NES code to address more ROM than a 16-bit memory address space would otherwise allow
2. Add additional "expansion audio" hardware, but only on the Famicom (not the NES)

Nestronic can play VGM files from games that use mapper hardware (which is most of them), because juggling all of that is done on the ESP32 side. What it cannot do, is play "expansion audio" generated by extra hardware those mappers may contain (e.g. VRC6), because it doesn't contain that hardware. In order to do that, I'd need to add an FPGA or a collection of salvaged mapper chips.

You briefly mentioned this above, but if I understand correctly NSF playback would require mapper hardware (or emulated) to process those 6502 instructions, but VGM playback already has them "processed" and is the recorded output from the mappers, etc?

Playing NSF directly on the NES CPU (2A03) would require emulating the mapper hardware, because playing NSF is basically just running NES code on a headless NES emulator. So you'd need to emulate the memory model that would be available to a NES.

VGM playback avoids this, because VGM is just a stream of NES APU (audio hardware) commands. So its already "processed" and thus doesn't care what the memory model of the NES looks like.

[sofakng]

Got it. I wasn't sure how the audio expansion chips worked and thought they might have generated additional APU instructions but it sounds like generated the actual sound which was mixed into the APU-generated sound.

I'm just curious (and again I realize this isn't directly related to the Nestronic so I apologize), but does the NES VGM format include any expansion audio data? If I understand correctly, this would have to be data captured from the expansion audio chips (ie. VRC6) and perhaps not possible?

[dkonigsberg]

it sounds like generated the actual sound

Yes, some mapper chips contain additional sound generation hardware. The Famicom had a connection to link it back to the console's audio output. The NES didn't.

does the NES VGM format include any expansion audio data?

It does. I've stumbled across VGM files from games that have it.

[sofakng]

Thanks so much for the help and the information!