search

frederic-mahe / Hardware-Target-Game-Database

Flash-cart optimized file and folder layouts.
GNU General Public License v3.0
651 stars 76 forks source link

NES and Famicom headers #174

Open frederic-mahe opened 4 years ago

frederic-mahe commented 4 years ago

https://nerdlypleasures.blogspot.com/2020/04/the-nes-and-famicom-accurate-cartridge.html

Use Great Hierophant's work to validate the N8 SMDB entries.

tjanas commented 4 years ago

One thing to keep in mind is that some features of the NES 2.0 header are incompatible with Analogue Nt Mini JB, and with regard to Nt Mini, sometimes a header value needs to be modified (e.g. to be able to change the palette on a VS. System rom). And Everdrive N8 doesn’t support NES 2.0 headers at all.

maniac79 commented 4 years ago

FYI: If you check the excel sheet from this blog entry you can find the same hash we have recently discussed for Mega Man 3 (USA) in column CRC32. Furthermore, he states:

CRC32 - Gives the CRC32 hash of the combined PRG-ROM and CHR-ROM (if any). This is NOT the CRC32 of the file, the header is not included in the hashing. PlayChoice-10 CRC32s include the content of the miscellaneous ROMs.

So, @frederic-mahe, I think this confirms your theory that No-Intro also stores headerless hashes and ROM Management tools ignore the headers too, resulting in the situation we face today that all files hashes in the SMDB don't match the No-Intro DB hashes.

In any case what @tjanas wrote above is critical and we have to be careful when deciding what to do next to ensure maximum compatibility. Btw, I'm shocked after reading that blog entry to say the least. Till yesterday I thought NES ROMs were more "stable", if that makes sense.

tjanas commented 4 years ago

Yeah, NES roms are a pain. FDS is worse because the disk headers can vary greatly between two pristine disk dumps (Disk Writer serial number, date values, etc). Plus the fact that save data is part of the disk itself. Also gets messy with FDS translations, especially if it’s a BPS patch that expects an exact checksum for the input FDS file. Many patches were written for old scene dumps of FDS that had bogus headers and unscrubbed save data.

SNES, Genesis, GB, GBA are much easier to deal with :)

N64 roms have its own unique quirks (BigEndian, interleaved, LittleEndian). I don’t have an Everdrive 64 but curious if it handles all three types.

frederic-mahe commented 4 years ago

So, @frederic-mahe, I think this confirms your theory that No-Intro also stores headerless hashes and ROM Management tools ignore the headers too, resulting in the situation we face today that all files hashes in the SMDB don't match the No-Intro DB hashes.

I wonder if we need to write a special version of the parse_pack.py script for NES files, one that would ignore the the first 16 bytes. It would solve the issue with hash values, but not the issue of header correctness and usability by our cartridge emulators.

frederic-mahe commented 4 years ago

N64 roms have its own unique quirks (BigEndian, interleaved, LittleEndian). I don’t have an Everdrive 64 but curious if it handles all three types.

@tjanas it is likely that the Everdrive is using an internal database of CRC values to decide what to do, with a generic heuristic for unknown ROMs.

N64 ROMs have .n64 and .z64 extensions. I wonder if the extensions have anything to do with these quirks.

tjanas commented 4 years ago

Should we adhere to the existing file extension standard?

https://en.m.wikipedia.org/wiki/List_of_Nintendo_64_ROM_file_formats

https://www.reddit.com/r/emulation/comments/7hrvzp/the_three_different_n64_rom_formats_explained_for/?utm_source=amp&utm_medium=&utm_content=post_body

I think Everdrive 64 supports Big Endian and it is also the format used by no-intro. Technically, that seems to be represented with a .z64 file extension.

https://datomatic.no-intro.org/?page=show_record&s=24&n=0717

frederic-mahe commented 4 years ago

Thanks for the links! I do learn a lot today :-) I moved that to a new issue.

steve1515 commented 3 years ago

I was trying to clean up some NES roms today and had all sorts of issues and then came across this issue.

It looks like there's a new database for NES roms that takes over from Great Hierophant's spreadsheet. See here: http://forums.nesdev.com/viewtopic.php?f=3&t=19940&p=248872#p248796

Lot's of info in Great Hierophant's blog post: https://nerdlypleasures.blogspot.com/2020/04/fixing-nes-headers-and-renaming-nes.html

Given that Great Hierophant now defers to the XML database (http://forums.nesdev.com/viewtopic.php?f=3&t=19940&p=248872#p248872), I believe that it should be the new standard going forward for NES roms.

tjanas commented 3 years ago

FYI, the pre-Noir Analogue Nt Mini jailbreak does not support full NES2.0 headers at this time. Until that occurs, NES1.0 headers offer the most compatibility. If some games absolutely require NES2.0 headers, then they can generally be supported by NTM JB if the last two bytes of the NES2.0 header are zeroed out. Also, the EverDrive N8 has some issues and known cases have been addressed (also documented by Great Hierophant in 2017).

tjanas commented 3 years ago

Can you be specific on which issues you are experiencing with the latest pack version and on what hardware?

steve1515 commented 3 years ago

I suppose if it breaks things then we should wait or possibly keep a 2nd SMDB for NES roms. I was under the impression that NES2.0 headers were backwards compatible, and if the rom required NES2.0 headers to work, then it wouldn't work on the N8 anyway with a 1.0 header. Out of curiosity do you have a link to the GH documentation from 2017? My searching didn't turn up any results.

As for the issue, I was having. It was that my NES2.0 header roms didn't match the SMDB. Which is ok, since the master branch contains an SMDB with 1.0 headers. I then noticed the NES_update branch which contains an SMDB with different file hashes. I had assumed this SMDB was an update for NES2.0 headers, but they didn't seem to match either. Do you know what the NES_update branch is for or why the hashes are different?

tjanas commented 3 years ago

The Analogue Nt Mini jailbreak will refuse to load a full NES2.0 header (meaning that if either the last two bytes are non-zero, it won’t load it). Also, the PowerPak doesn’t support NES2.0 headers at all.

I made the changes to the NES entries in the pack with a detailed changelogs of how headers were changed. The NES headers have always generally been legacy NES1.0 headers. My changes have made them more correct NES1.0 headers, except for the few that have mappers above 255, I’m which case I used NES2.0 headers compatible with Nt Mini JB.

The branch you should always reference is master.

tjanas commented 3 years ago

Here’s one of the changes I made to the master branch in September in regards to NES headers: https://github.com/SmokeMonsterPacks/EverDrive-Packs-Lists-Database/pull/399

tjanas commented 3 years ago

Nt Mini Noir SMDB was created yesterday that uses verified full NES2.0 headers.

If you have an original N8, feel free to test some of these. I do remember running into issues with some games where the mapper and/or other header flags had to be changed for N8 compatibility.