[Proposal] - stablish a meaningful rom/game-media format.

[MarcoEstevez]

Hi there.

Recently I got inspired by ePub files ( an e-book format for small devices ), basically is a zip file (if you change its extension you will be able to open it like a folder ) containing the TOC and pages in standard html and css files, it also has the book cover as an image usually jpeg, and an XML file containing lot of Metadata, like author, publishing date, publisher, synopsis, genre, sub-genre, etc. Since the "book" file self-contains all this extra metadata, as long as I throw them into any eReader software or device, those are able to automaticaly categorizie, and visualize properly among rest of stuff, creating grouped and ordered lists, or nice collection tiles.

After that I though same thing could happen for roms and emulators, If there were a standard rom/game-media container, with metadata, any frontend/emulator could benenfit by sharing a common API and also avoiding the need to handle separately box-arts, Snaps, names.

For instance, now a days to collect any rom info, it has to be compared CRC files against a database and from there collect rest of media and download it, or pre-download all media files for an specific set/catalog and then match them at realtime.

---

Please notice, I'm not asking for any implementation but just an specification to try to stablish an standard container for videogame-media (roms) as already exist others ones for music, books and videos such as mentioned ePub, or MP3 for sound files. Thus once established, any emulation software could implement and be free to use among all existing roms.

That way people could put their dumped roms inside that container, together with its boxart, scaned - instruction boklet, metadata, etc and hopefully it could contribute (if the idea succed) to preserve videogames in a better way, and not only the software.

What do you think about it guys?

[hizzlekizzle]

It's a good idea, but it's one that many people have attempted over the years and never gotten any traction with. byuu's "foltainer" format is but one example.

[MarcoEstevez]

That's why I came here, Retroarch is the biggest player :-), and covers almost every retro system and has the potential to explode all benefits thanks to the neat and powerful GUI

here a quick draft, just imagine could chose an specific rom-hack or translation, or apply cheats on the fly, or do pause on retroarch to check the official manual or world map picture for old lovely 16 bit RPGs all out of the box, or play the official game trailer for nostalgia.

    some_file_name.xxx
    ├─ media
    │  ├─ cartridge.png
    │  ├─ banner.png
    │  ├─ front.png
    │  └─ back.png
    ├─ instructions
    │  ├─ booklet
    │  │  ├─ 001.png
    │  │  ├─ 002.png
    │  │  ├─ 003.png
    │  │  ├─ ...
    │  │  ├─ ...
    │  │  ├─ ...
    │  │  └─ 00n.png
    │  ├─ sweet_word_map.png
    │  └─ any_other_propaganda.png
    ├─ cheats
    │  └─ cheat.cht
    ├─ mods
    │  ├─ game_hacks
    │  │  ├─ great_hak_001.ips
    │  │  ├─ great_hak_002.ips
    │  │  ├─ ...
    │  │  ├─ ...
    │  │  ├─ ...
    │  │  └─ great_hak_00n.ips
    │  └─ fan_translations
    │     ├─ trans_001.ips
    │     ├─ trans_002.ips
    │     ├─ ...
    │     ├─ ...
    │     ├─ ...
    │     └─ trans_00n.ips
    ├─ metadata.xml  ( game related data )
    ├─ manifest.xml  ( summary of the content )
    .
    .   optional stuff
    .
    ├─ soundtrak if the game came with it
    └─ game_official_trailer

[ghost]

Yeah, imagine doing this to all supported cores, platforms or systems, and thousands of games in rom or cd.hdd format. we dont even have a proper search function yet for the playlist, let alone able to sort stuff by meta (or something). wanna start sending PR(s)?

[MarcoEstevez]

As I said, this is just a proposal of intentions, one specification, nothing else, I wanted to know what is your oppinion on this.

In an early adoption, and in order to minimally support it Retroarch just need read the rom inside a zip, what I believe already work for most of the cores... well technically it should read the manifest document to collect what the rom is, cause the zip will contain other stuff.

Once this point is reached, other planed features you mentioned, like sorting, searching, grouping, organizing, could become more easy, since all the information is in game_file itself avoiding extra jumps to the database files.

For instance, if were ever implemented a feature that allows creating a list with all rpgs among my library, right now you need to go throught every game and then jump to its respectively rbd (datafile) and query there (Asuming we do have that info), but having this info on every game-manifest, that task could become trivial.

[i30817]

Not to rag on this, but not only this is a significant endeavor to have the 'good features' (one i'd love to add to chd for example are 'reversible hardpatches', a hardpatch that keeps a reversal patch concatenated to the file), it's actually unnecessary for retroarch to present multiple cds/floppies as a single game.

I've opened a issue delineating a strategy that accomplishes that part for RA : https://github.com/libretro/RetroArch/issues/6410

If you're stressing about the 'actual' filesystem, my experience is that you don't want a single file per game if the game is in multiple roms.

Not only it ruins checksum metadata, not only emulators handle that case badly in other ways, not only compression formats are terrible about compression indexability of single files, much less multiple; you can never tell what kind of file in a 'collection' will need a update. Rom files, walkthroughs, cheat files, patch files, hacks etc.

Not to mention that trying to branch out that to 'all kinds of roms' is asking for trouble. For instance, did you know some 'roms' for cores in RA aren't actually 'read only'?

Just keep games in their own directory, unzipped, with external filesystem compression if you need it. It's getting to the point that if you add another layer compression lusers and emulators would have to deal (badly) with 4-6 layers of compression (original fileformat compression like TIFF, then game filesystem compression/encryption like PSP games, then your idea of 'fileformat compression', then a idiot user placing that in a rar, then the idiot user placing a rar in a 7z, then placing the resulting abomination on a NTFS compressed folder. Add ecm for ps1 games somewhere around this stack).

Dedicated compression fileformats do have advantages (.chd has the advantage of having a unique 'data checksum' which is only one regardless of the number of original files and ability to concatenate metadata without affecting that internal checksum) but the libretro-database is not ready to take advantage of that because they don't have all the set games to convert to .chd and record the checksums.)

And i'd leave the fileformat specifications to the actual specialists. CHD is doing a good job and support for it is spreading among among RA CD cores. It's only the 'other files, multiple CDs' support it doesn't have and as i mentioned, those are bad ideas anyway (imo) but the 'other files, manuals etc' could be 'added' well enough by RA because of its metadata functionality. As long as the data is usable by RA that is, which i'm not confident of and the dumper projects care to change formats and dump manuals, which i'm certain won't happen because it's a lot of work.

[i30817]

That said, if the 'compression' part is taken away, i can see a good usecase for having a 'shadow' filesystem (as a directory) that has several RA features which need files enabled by the right dir and right named files existing.

Your banner and cheat example for example, would be a good way to only download what's 'needed' if the RA downloader had that granularity.

If that shadow filesystem needs to exist on the same dir as a the ROM or on a child dir, or even on the RA system dir, i'm agnostic to.

RA already downloads/updates system metadata, but it's a brutal amount, individual downloads could make it more scalable for updates and broaden the files to other kinds of metadata/extra data.

That said, there is a reason that RA uses zip for the banners. If you happen to have a whole set, uncompressing that zip is worse. Mmmm...

[i30817]

I also agree it would be nice to have the possibility of having multiple softpatches on a ROM dir. Currently when you softpatch, you have to have 1 hack the same name as the rom dir and in the RA playlist (after using the RA scanner) it has the name of the original rom, because ofc, that is the CRC it detects. A way to associate several ips/bps to a single rom and present them as differently named games would be nice. Currently i duplicate the directory/rom and edit the playlist entry to the hack name but that's unneeded work and space wasteful if your filesystem doesn't have deduplication.

Speaking of softpatches, a way to apply many of them in order to the same rom would be nice too; not only one of many to the same rom.

That said, i'd live with a VFS that just presented the set of {ROM + ips} as a single file for checksum purposes (or copy-on-write), since libretro-database seems determined to categorize hacked roms. I much prefer softpatches for updates (i keep reversal patches for cds, where softpatching is not supported). It wouldn't work to present metadata for hacks that are joinings of multiple individual patches, but nothing does anyway.

[MarcoEstevez]

Thanks for your reply, I never mentined compresion as a desired requirement or anything, I said zip cause is an easy way to "pack" diferent files with the external apearance as an atomic and protected piece easy to handle (from an end user perspective), so I'm fine using VFS or simple OSs filesystem (normal folders).

So having a VFS with a convenient structure, might be also beneficial in the future, cause you might gain some conmon features across different systems, for instance perhaps some parts of the cores can be taken away and move them out to a common unique API. not to mention scan RA logic will get simplified a lot dealing just with a single identification method.

[i30817]

Unfortunately, RA already uses a single identification method. I say 'unfortunately' because there are plenty of usecases for fuzzy ids. And because this forces users of cores that don't play ball (like scummvm) to introduce junk files just for the scanner.

This method is checksums of the whole rom index file¹ on the playlist. It works well when the roms are contained in zips because zips pre-calculate CRC32 and can present it without the emulator having to recalculate it, and works horribly (for the scanner) for plain index files that happen to be huge like isos. This actually could be improved by having code abstracting querying 'optional' filesystem metadata because some filesystems actually have checksum information recorded. But no one is willing to do all that work (except Java 8 apparently iirc).

RA is already (trying) to move code that can be moved out of the cores. Softpatching for example, falsely advertises that it's 'common' (there is even cmd line switches on the RA executable to enable/disable it regardless of core), but doesn't actually currently work for all cores.

The hope is that the new VFS will support this better regardless of ROM size, along with other useful hacks for some cores (for example, it would be good for the computer cores to enable a copy-on-write layer 'above' the layer the scanner scans, so the original data is protected and the scanner only ever uses a stable checksum as primary database key).

¹(also there are some bugs because the checksum is not unique per game because RA only checks/uses the 'index' file checksum and on some cores that checksum is not unique - this is why CHD internal checksum type is superior to even zip).

[MarcoEstevez]

Well at some point, as you mentioned, RA have to decide whether scan the file to get a checksum or read it directly from a the zip header (extra logic).

To avoid this, can be added to the specification that whenever a new VFS is created or modified the checksum have to be calculated and stored there, thus RA has no longer to keep any logic for the scanner.

[i30817]

That extra logic is like, 5 additional lines. What's actually sad is that they junked the support for serials instead of extending it and using it as a fuzzy game specific cfg id that works across several dumping groups 'standards'.

I'm not against a record of the serial/checksum, but i don't think RA devs will agree because they thought a checksum was a appropriate primary key and only relaxed calculating it in cases where the normal user can't write gibberish to it.

They'll say that it's already on the playlists (in spite of the playlists not being portable catch22). Another possibility if put them on the .cfg files for that game, but since i was hoping to use that one of the primary keys for finding cfgs, probably not a good use to 'assume' it's correct.

I actually think that there should be yet another method to search (CHD internal checksum), because it solves real problems with some dumping groups 'standards' (i'm looking at you TOSEC dreamcast that forced the dreamcast set to check for track1 checksums instead of .gdi) and would enable a good compressed standard to replace zips. RA devs don't have fullsets to make this database though.

I also appreciate that RA is trying to get a unique id to all roms but it's kind of doing it wrong imo. Several dumping groups just take roms from others and rename them and end up with the same checksum. It would actually be more reliable to check for checksum and name to fetch metadata (even if it's equivalent except for name in those duplicates). I get that it's supposed to 'work' for dumps where the users renamed the file, but i find a goodsnes file on a no-intro playlist much more distressing than being forced to use clmamepro to rename files.

[orbea]

While I am not really opposed to this idea, it seems like a lot of effort for not that much reward.

[merlink01]

+1 for this Idea. If you look at Kodi Media Player it has the possibility to use a standartized directory structure. All Metadata and pictures can be saved there at specific places. Also it's possible to zip this whole folder, an Kodi uses it as a folder using a vfs. With games and all the possible extra files this makes a lot of sence to me.

[andres-asm]

I think this is like shooting yourself on the foot. It's a good idea, never gets enough traction and you end up with yet another standard. Ask byuu.

[merlink01]

So it would be nice to see the other 14 standards for organizing Roms. But I cant find 1 at the moment. I tested scummvm in retroarch and the message was: No data foud After some serarching I needed to add a .scummvm file to the directory and put some shortname of the game inside this file. Is this user friendly? But if i have to do this, why not write an .nfo file that could hold this information + other metadata that are usefull? If the scraper finds some more information why not simply write these to the xml. Also the crc checksum could be included, so the scraper must not work hours for hundreds of psx games as an example.

[Ferk]

Since Kodi is adding support for libretro, it's most likely that they will use .nfo files placed next to the game to store metadata, just like they do for all the other media libraries. Though they don't have support for it yet (there won't be a game library in Kodi v18 yet by the looks of it).

Related discussion in the Kodi forum: https://forum.kodi.tv/showthread.php?tid=268963 A developer of MediaPortal-2 showed also interest in collaborating, so it would be great if Retroarch followed the same format too.

Retroarch's approach of having to download entire artwork libraries and databases of metadata for all the games ever released for each system does not scale, and it's not even possible for some of the cores, or with homebrew content. I do find Kodi approach with scrappers and local nfo files more sane. I know the scrapping would be too complex to add at this point in Retroarch, but imho, being able to read local nfo files or similar to construct the playlists and set the artwork would be good enough (no need to even add fancy filters or sorting, imho).

Once Retroarch is able to use the metadata offered in the directory of the game itself, then packaging the directory into a container/archive where it can be mounted from VFS should already be a good enough of a format to distribute games with. No need to specify complex internal directory structure.