[MAME] Adopt Non-Merged playlists

markwkidd commented 8 years ago

I know that there has been a lot of activity and discussion about the database, and that some entries have already been added for MAME and FBA. I would like to make a heartfelt request, like when I posted about the Stella / Atari 2600 database, to either add to or replace the existing Arcade database entries. As with the Atari 2600 ROMs, I'll try to describe the reasoning for this request:

Whatever arcade sets are preferred should be complete and as readily available as possible. Complete non-merged sets for FBA 0.2.97.38 and MAME 2003/MAME 0.78 are readily available and in fact come up higher in search results than the Split and Merged sets in circulation.
Non-Merged sets are easiest to work with for users who do not want to install a complete FBA or MAME set -- probably most users. Non-Merged allows them to add an individual game ROM without having to worry about tracking down its parent file as in a Split set.
Non-Merged sets, like the preferred No-Intro and Redump sets, represent ROM files that are fully intact from an archival standpoint. Split and Merged sets are basically hacks to reduce bandwidth consumption at the cost of the archive integrity of clone/child ROMs.

It seems that Non-Merged MAME 0.78 with the 2003 core is already officially recommended MAME set for RetroPie. In case it would help explore this issue any further, I generated a DIR2DAT for the MAME 2003 Reference Set (non-merged) ROMs, including CHDs: MAME 0.78 Non-Merged with CHDs - No BIOS.txt (change extension to .dat)

barbudreadmon commented 8 years ago

Split sets also help to reduce disk consumption, having a complete non-merged set on my raspberry would be a serious drawback over the current split one.

Btw, what comes first in search result is irrelevant, anyone can download clrmamepro and learn to use it within 10 minutes.

markwkidd commented 8 years ago

Relevant questions:

Is it within the vision for RetroArch for expect users to learn ClrMamePro? (I am very comfortable using it, but I wouldn't wish it on anyone else)
What percentage of potential users are likely to want an entire MAME set?
How much size difference is there really between the Split set and the Non-Merged set? I have found only a 25% space savings for the complete MAME 0.139 set with CHDs between the two types. Are you actually sure that it would make as much difference on your Raspberry?

barbudreadmon commented 8 years ago

People who want to play arcade games should learn ClrMamePro, or they'll never understand why their rom won't load and will endlessly come opening issues like "my mame rom is not working on fba".
I don't know, i'm one, i know others.
From my perspective, when you are on limited storage, every bytes count. 25% of 12Go is 3Go, that's worth about 10 psx games.

markwkidd commented 8 years ago

Browsing the RetroPie forums it seems that Non-Merged MAME 0.78 with the 2003 core has become the officially recommended MAME set for RetroPie. I do hear that there are users who want the space savings of a split set, but if there was general concern about non-merged sets from users with limited storage, I think that is the place where we would see it.

In case it would help explore this issue any further, I generated a DIR2DAT for the MAME 2003 Reference Set (non-merged) ROMs, including CHDs: MAME 0.78 Non-Merged with CHDs - No BIOS.txt (change extension to .dat)

cudencuden commented 7 years ago

fyi, FBA 0.2.97.39 is out

markwkidd commented 7 years ago

User having trouble making the transition from RetroPie to RetroArch due to this issue for the MAME 2003 core: https://www.reddit.com/r/RetroArch/comments/5dx3v5/retroarch_wont_load_078_mame2003/

lasers commented 7 years ago

(A)

MainMenu->OnlineUpdater->Database-Updater->MAME (Merged)
MainMenu->OnlineUpdater->Database-Updater->MAME (Non-Merged)
MainMenu->OnlineUpdater->Database-Updater->MAME (Split)
MainMenu->OnlineUpdater->Database-Updater->...

I think people wouldn't enjoy downloading dats individually, but...

MainMenu->OnlineUpdater->Update Database->(all excluding MAME dats)
MainMenu->OnlineUpdater->MAME-Updater->MAME (Merged).dat
MainMenu->OnlineUpdater->MAME-Updater->MAME (Non-Merged).dat
MainMenu->OnlineUpdater->MAME-Updater->MAME (Split).dat

Would that work well for everybody here wanting different dats?

(B) I never played with MAME so I don't know if this is possible or would work... Make a separate dat / playlists for MAME especially if size / crc / md5 / sha1 are not same? We may have to use three identical MAME cores like this MAME - Non-Merged.. kinda like how we already have one Nintendo - Super Nintendo Entertainment System with lot of different cores... only this time... Three systemname with one core (or three cores).

What would work well?

barbudreadmon commented 7 years ago

Seems to me like this is the wrong way to solve this :

Those multi dats will be hard to maintain.
Having only one dat, and have the scanner check the zip for the files listed then the parent zip file for missing files would cover all cases, this is how those cores load a rom.
Scanning the inside of the zip file would also make more sense because sha1/crc of a zip file containing multiple files is random with normal zip software, and people don't understand why the full romset they built through clrmamepro won't be found by retroarch's scanner.

Upgrading the retroarch scanner in such a way is impossible ? If so then i think the second best solution would be to still have only one dat file, but without sha1/crc, scanning only the name of the zip files and not caring about the content.

markwkidd commented 7 years ago

Split and Merged sets are purely a file compression approach meant to economize the transfer of ROMs over the internet in the early 2000s. Moreover it's a file compression approach that is pretty much unique to arcade ROMs.

RetroArch doesn't support .7z or .rar compression schemes, or other esoteric compression methods. It seems an odd exception to prefer Split sets for the arcade database.

Non-Merged sets are the equivalent of headerless ROM sets in the cartridge emulation world -- it is a complete dump of a game and the most accurate representation of the source data. Non-Merged sets are readily available for the historical libretro MAME cores and can also be generated from Split or Merged sets by widely available software.

(That being said, @lasers idea is kind of close to my own thoughts about alternative ROM scanning methods: https://github.com/libretro/RetroArch/issues/2033#issuecomment-234686250 )

RobLoach commented 7 years ago

Adapting TorrentZip for the sets will give us consistent CRCs for the sets, since TorrentZip generates the same zip files across all platforms. The following is an example of using it with Pac-Man:

trrntzip pacman.zip 
Skipping, already TorrentZipped - pacman.zip
Execution time 0 hours 0 mins 0 secs

Checked 1 zip file.
  1 file was already up to date.

Size: 28432
CRC: ee16c335
MD5: e36e28c1e3be1dd8811f2c43a6191899
SHA1: dcb0bbc7e7fa5a22269715b79feb9140d4ee1b05

Could do the same for all the formats: http://romulus.net63.net/tutorial/tutorial.html

I don't have a MAME collection, or at least a recent one, so we'd need someone who would do this for us.

RobLoach commented 7 years ago

@markwkidd Finally had a chance to test this out a bit. Nicely done! Upped a way we could split the databases per core over at https://github.com/libretro/libretro-database/pull/460 . Would appreciate a review. Thanks!

markwkidd commented 7 years ago

Hooray, this happened! I'm closing the issue.

libretro / libretro-database

[MAME] Adopt Non-Merged playlists #248