ideas for a better hiscore file format

[eadmaster]

An ideal file format should:

• include the addresses the memory was dumped from, so the hiscore can be loaded even if the hiscore.dat is missing/obsolete/etc.;
• have an internal integrity check;
• have an header;
• be human-readable.

solution 1 using json:

{
    "supported_content" : [ {
            "system" : "nes",
            "content_name" : "Super Mario Bros. (W) [!]",
            "content_crc32: "3337ec46"
        }, ...  // more games can follow
    ],
    "content_memory_patches": [ {
            "cputag": "maincpu",
            "addressspace": "program",
            "address": 0x7df,
            "length": 0x04,
            "first_byte": 0x00,
            "last_byte": 0x00,
        }, ...  // more patches can follow
    ],
    "hiscore_data": "AAEHAA==",  // base64-encoded or hexdump?
    "hiscore_data_crc32": "6fc723ec"
}

solution 2 using existing binary file patching formats:

• IPS -> NOK (max size limits)
• UPS
• xdelta (has input verification, has ability to scan for data location changes)
• bsdiff

Unlike the json solution they are not easily editable and could lead to confusion since they are often used for static ROM patching as well.

[DreamPlusInfinity]

I suggest the following.

Addition of new entries to the format

Type

• Type 0: Direct value. The entirety of the score is converted to a hexadecimal value and is stored in the game, then converted back whenever displayed. Ex. Score 9999 is stored 0x27 0x0F (or 0x0F 0x27 depending on endianness)

• Type 1: Each 2 digits is stored in 1 byte. Value is converted from hexadecimal to decimal then displayed. Ex: Score 99 is stored 0x63.

• Type 2: Each 2 digits are stored in 1 byte. Value is stored in decimal, and displayed directly. Ex: Score 99 is stored 0x99.

• Type 3: Each digit is stored in 1 bye. Value is stored in decimal, and displayed directly. Ex. Score 99 is stored 0x09 0x09

There's potentially more types, to be added and supported as they are encountered in games.

Order

• Least significant byte in the score value first
• Most significant byte in the score value first

Multiplication Factor Some games add extra zeroes to the right, but they're not actually included in the internal score tally.

Improvements to existing entries

The CPU memory format may be insufficient for consoles with bankswitching. I recommend adding an optional entry for banks.

Exporting Scores to Leaderboards / Updating Hi-Scores Ingame

I suggest adding for each game an optional list of preset events. The events will be memory offsets (space, address, bank) to ASM, and will have watchpoints tied to them that will trigger actions.

• Begin Player Session: The game's "RAM Score Offset" is constantly copied to an emulator variable called "RA Current Session Score", as long as the session is active.
• Abort Player Session: The above tracking is stopped, and "RA Current Session Score" is discarded.
• Conclude Player Session: The emulator stops tracking "RAM Score Offset". The value in "RA Current Session Score" is compared against 3 leaderboards stored by RA: "Game's LB", "Local LB" and "Global LB", and will display if the player is out of rank, or just got rank X, or got the highest score. The score is compared against a fixed hiscore value for a doable completionist challenge (like 99,999 in Panel De Pon) and if it's greater or equal to it, then the session is flagged "Perfect".
• Update Global LB: To minimize network usage, global leaderboards will be only read once, when the game ROM is first loaded (and if the option to sync to global rankings is enabled). If some conditions are met (not an "improper" flagged player session, your score is greater than the current lowest score in the loaded copy of the Global LB) then your score is submitted to global leaderboards. If accepted, you get your global ranking, and the current Global LB is loaded again once to local storage.

Sessions may be flagged with these flags, which will appear next to scores in listings:

• 1CC: Finished from beginning to end without game overs.
• Clear: Finished the game, but got at least one game over / continue screen.
• Perfect: Beat a fixed hiscore value for that game.
• Improper: Used emulator cheats, ingame cheats or unapproved rom versions. Rejected from Global LB.
• Casual: Used inaccurate emulator enhancements, like input lag reduction, overclocking, enabled Up+Down...

Leaderboards may be three lists. Might be good to look into separating leaderboards further to separate difficulty settings.

• Game's LB: Uses the ingame hiscore, or hiscore list. If the game has no backup memory, then it uses whatever presets it has on startup for the hi-score(s).
• Local LB: Different leaderboard, starts out blank. Tracks out the current user's best scores in RA with that game. Includes date and time.
• Global LB:

Now for the events and associated actions whenever their execution watchpoints are tripped:

1. Game loads Scores from Save Data (SRAM) to "RAM Game Hi-Score". 1'. Game loads preset score to "RAM Game Hi-Score" Action: If first time only, Update "RA Game LB" with RAM Game Hi-Score(s). On subsequent times, overwrite RAM Game Hi-Score(s) with "RA Game LB" contents (might be better to make this overwriting optional and up to the player)

2. Initiating New Game. Action: Begin Player Session. Flag Session Flag as 1CC.

3. Initiating Attract Mode. Action: Abort Player Session.

4. Game Over / Continue Screen Action: Conclude Player Session. Session Flag 1CC disabled.

5. Choose "Yes" on Continue Screen Action: Begin Player Session.

6. Ending (may be multiple) Action: Conclude Player Session Session Flag Clear enabled IF 1CC disabled.

7. Accessed ingame cheat codes / debug modes Action: Session Flag "Improper" enabled.

Fallback Events and Actions:

1. Current Score is zeroed out (only if the above aren't specified) Action: Disable all flags, except "Improper" if enabled. Conclude Player Session.
2. Game Closed / Game Reset / Emulator shut down Action: Disable all flags, except "Improper" if enabled. Conclude Player Session.
3. Emulator Cheat Codes enabled Action: Enable Flag "Improper".

[eadmaster]

Addition of new entries to the format: Type field

this could be useful to keep hiscores with games with no native support (https://github.com/eadmaster/console_hiscore/issues/2), but i think it would be better using string constants to describe them (e.g. lsb_value, msb_value, etc.)

Improvements to existing entries The CPU memory format may be insufficient for consoles with bankswitching. I recommend adding an optional entry for banks.

So far i've found no problems with NES games using bankswitching as they are mapped automatically by the emulator cores as "virtual" main memory.

Exporting Scores to Leaderboards / Updating Hi-Scores Ingame I suggest adding for each game an optional list of preset events.

I think this exceeds the aims of this project and also overlaps with RetroAchivements.

I'll try to keep things as simple as possible here so it would be easy to re-implement hiscore support in other 3rd party projects (emulators, flashcarts, etc.).