Feature request: Standardized method of integrity checking support files

[Shoegzer]

As discussed here, RA currently integrity-checks support files through MD5 hashes in core info files. This is inconsistent between cores and some of these are based on zip collections rather than their constituent files as they should, while others are based on individual files. Still others don't perform any checks at all. Among other drawbacks, this can lead to false RA or core reports from users who simply have missing/bad/corrupt/etc. files.

Ideally RA itself could perform these checks, perhaps against an internal database or xml file containing hashes of every support file needed by each core. Such an approach would be consistent with MAME's high-quality standards for maintaining such integrity and assist in efforts to preserve those files properly as an ancillary but important side-effect.

[RobLoach]

I'm assuming it's reicast you're having BIOS issues with?

The reicast_libretro.info file covers the files it checks for. The md5 file it checks for is torrentzipped of the expected naomi bios contents. reicast expects a .zip file, so we Torrentzipped it, as that produces the same file if the contents are the same.

[Shoegzer]

Reicast is one, but this for a standardized method of handling not only that core, but all others current and future. Also, I understand why it was torrentzipped; I'm suggesting there should be a better way to integrity-check files which includes checking each file within the zip, in a similar manner to the method employed by MAME since it's inception. Hashing a torrenzipped file is also suboptimal for other reasons described in the link above.

[RobLoach]

On the libretro-database front, we've been somewhat maintaining BIOS.dat to help keep validate the files. It's missing a few files, which we'd love help getting in there.

[Shoegzer]

Ah, yes! That's exactly what I had in mind. Is the plan for this dat file to replace hashes in info files? If so, I think that would nail it. Also, don't forget to update it with this. ;)

[andres-asm]

To be completely honest I don't like that approach either. What should happen is that an API extension should be created so that on load core an environment callback should run, informing the frontend of the required BIOS files. This structure would contain information about the BIOS files and the filetypes affected by that BIOS file.

For example (very rough pseudo code)

libretro.h

struct retro_firmware_info
{
   /* pipe separated list of the lookup locations */
   const char *desc;

   /* A computer friendly short string identifier for the system.*/
   const char *ident;

   /* Filetypes affected by this firmware file. */
   const char *valid_extensions;

   /* Filename */
   const char *filename;

   /* Buffer to pass back the location (not entirely sure of this part) */
   char *location;

   /* CRC32 of the supported BIOS file */
   unsigned crc32;

   bool optional;
};

Core:

char bios_location[PATH_MAX_LENGTH];

static const struct retro retro_firmware_info[] = {
    {"Sega CD|Mega CD|Whatever Other Name", "mcd", "cue|toc", "segacdbios.bin", NULL, , 123456789, false},
    {"Sega Master System", "sms", "sms", "segamastersystem.bin", NULL, , 223456789, false},
    { NULL },
}

Then an environment callback would set this, and now the Frontend would look for these files in: This should happen on retro_init.

`%SYSTEM DIR%/ %SYSTEM DIR%/Sega CD %SYSTEM DIR%/Mega CD %SYSTEM DIR%/Whatever Other Name %CONTENT DIR%/ %CONTENT DIR%/Sega CD %CONTENT DIR%/Mega CD %CONTENTDIR%/Whatever Other Name

For Sega CD, and once it finds it back it would feed it back to the core using another function. Of course this is pseudo code but I think somehting like this should be done.

Looking up the files in other locations is up to the core, you could add more dirs that would be used as root locations. The important part is that the frontend (that is more flexible and knows more about the filesystem and has lots of helper functions) would locate the file and tell the core where to find it.

If it can't be found (this was done on retro_init) we can fail load_content preemptively (depending on the optional status of the bios), and show messages completely in the frontend side of things.

The core knows what it needs and it is ALWAYS UP TO DATE on it's own requirements so this is the only long term viable solution.

Any solution using external files would be a waste of time in my opinion.

[andres-asm]

Of course libretro is now implemented in more frontends so this would be optional, if the core receives no BIOS location then it should look for it by itself in system dir

[RobLoach]

We have no plans to use BIOS.dat in RetroArch, but we do hope to keep it synced with what's in the core .info files. Open to contributions!

[Shoegzer]

Hmm.. There are already other databases such as MAME itself that keeps tabs of good dumps and hash files. Wouldn't this then be duplication of effort if it's not used in RA? Maybe I'm missing something.

[andres-asm]

Using external files is always a problem. The core knows what it needs, it always does. MAME doesn't really count since "BIOSES" are just parent roms and it's not something that can be achieved with either method.

For MAME/FBA what needs to happen is that retro_load should fail and show a message in the GUI.

[Shoegzer]

@RobLoach I should just ask.. what is the point of maintaining BIOS.dat if it's not used? Also it just seems to me that manual syncing between BIOS.dat and core info files is prone to error over time as they'll fall out of sync and with eventual lack of resources/interest.

@fr500 I see your point and agree that a solution based on an API extension seems elegant. Although, external files invite more opportunities for others to contribute updates vs. "hiding" metadata in the codebase which is generally less accessible. MAME itself does both actually - an internal database for hardware ROMs (used by -verifyroms), and a series of external XML files for software. These XML files not only guard against "code bloat" in the MAME source, but also - again - has permitted many non-devs to contribute critically-needed updates. Such a solution is vital in the case of software, especially considering how many consoles and computers MAME supports now - though the use case is admittedly a bit different from RA's.

Either way though, RA badly needs a standardized method of checking basic system rom integrity - hence the point of this feature request.

[RobLoach]

what is the point of maintaining BIOS.dat if it's not used? Also it just seems to me that manual syncing between BIOS.dat and core info files is prone to error over time as they'll fall out of sync and with eventual lack of resources/interest.

Correct, it's prone to human error, and things can get out of sync. Having BIOS.dat allows verifying your BIOS files with clrmamepro, or a similar program.