MAME 2003-Plus database update from Nov 27, 2018 regression (?)

[metchebe]

The latest MAME 2003-Plus database update from Nov 27, 2018 makes Retroarch leave out some of my roms. The previous version (initial database) worked perfectly.

For the valid ones left out, torrentzipped file CRCs are :

Filename	File Size	CRC32
aburner.zip	717.919	fcb29e7e
finalap2.zip	2.167.299	5ada7258
pacman.zip	10.925	368f99b9
simpsn2p.zip	251.134	d7829dd9
tetris.zip	138.007	22ff0eac

Note: romsets validated with latest core-generated DAT from today (Nov 28, 2018), in Romcenter.

[RobLoach]

@markwkidd Do you know if some of the removals were undesired?

[markwkidd]

@metchebe thank you for posting about this. I'm trying to make sure that the playlist scanner only supports romsets that are actually playable in the core. My first attempt to provide a playlist scanner database definitely included some romsets that were not working due to a bug in my script so it's important to get this feedback in order for me to keep fine tuning the list.

That being said, most of these romsets have either also been updated since your romsets were built (not possible if you validated with the latest DAT) or you need to run TorrentZip on your romsets because there are different CRCs.

Here is what the scanner is currently looking for on each of these romsets:

• rom ( name aburner.zip size 1517730 crc e8c1d4e6 md5 16da68d929814efc09d5d238c9f134a4 sha1 6a0b9924c30b4c8fa9962e3bd647409eb76789dd )
• finalap2 is flagged in the MAME 2003-Plus driver as GAME_NOT_WORKING, so it was intentional to remove that from the playlist scanner. It is certainly possible that the flag is not correct and that the game can be played through. If you think this romset is actually working, then we should talk about removing that flag from the driver.
• rom ( name pacman.zip size 14145 crc aaa1920a md5 c95dd87735185955b9452d09580c2bfc sha1 5c4f678119d9cbc9fc4e4b72d4f9951871273c6a )
• rom ( name simpsn2p.zip size 2486348 crc 3ffdd78e md5 2a74b3128cf6c05441fd83bb445bb091 sha1 037b6e648c71052e802155ac5448412eff42b366 )
• tetris - as far as I can see, this one has been mistakenly removed from the DAT by my script. Dang it, everything else makes sense on your list but this one does seem like a bug with my work.

[markwkidd]

OK, I have found the reason why tetris was removed although there is seemingly an issue I'll need to resolve. Our catver.ini indicates that the tetris romset is unplayable: https://github.com/libretro/mame2003-plus-libretro/blob/master/metadata/catver.ini#L4276-L4277

That being said, something is wrong because the category list shouldn't have Unplayable for a romset that doesn't also have a flag on its driver. tetris doesn't have a flag on its driver, so I need to make a fix either to the driver or the catver.ini.

I'd guess that it's more likely that the catver.ini is wrong and needs to be corrected but I would like to be sure -- @metchebe have you played the tetris romset much in this core? Do you know it's in good working order?

[metchebe]

I obtained the following roms from the MAME 2003 from the reference set: aburner.zip, pacman.zip, simpsn2p.zip. They indeed do match the CRCs the database has. The difference that they have with my files is that they include parent roms (checked with RomCenter, note the bigger file sizes). Maybe before there were CRCs for merged and unmerged (?).

Final Lap 2 has never worked well, so it is reasonable to exclude it.

I can check later if Tetris works.

[metchebe]

tetris Tetris (Sega Set 1) indeed does not work.

MAME recommends using working clones tetrist or tetrisbl. If they use the same driver, that would explain why it is not flagged.

[markwkidd]

@metchebe thank you again for helping me as I continue to tidy up -- but also better understand -- the metadata for these drivers. As far as I know for sure, you and I are the only ones using this scanning database yet.

As an aside, you may also find utility in the mame2003-plus thumbnails repository I have been compiling: https://github.com/markwkidd/mame2003-plus-thumbnails

To recap about the scanning:

• tetris and finalap2 are correctly being removed from the scanner due to GAME_NOT_WORKING flags
• aburner pacman and simpsn2p are not matching your romset zips.

Let me mention something that you may already know. The RetroArch playlist scanner is using the CRC of the romset zip itself (like aburner.zip) to match it to the playlist. This scanning approach is different than MAME's own matching process, which looks at the files inside the archive.

Therefore, in order to match the scanning database, the romset zip files must be exactly the same as what the scanner is looking for. Does RomCenter have equivalent settings to the Full Non-Merged settings in ClrMamePro?

For example, Here is how the Full Non-Merged contents of pacman.zip should look after being processed by TorrentZip:

[metchebe]

I see now that I have always been using the 'split' option in romcenter (reference doc):

Split: common roms between a main and its clones are stored only in the main game and not in each clone file

The database is OK for these roms in 'unmerged' mode. Split clones indeed work with the older database, so for me it was a regression. Anyway, I suppose it is up to the team to decide if split files are kept.

For me the issue is resolved as it is a design choice, thank you.

[markwkidd]

The core itself does support Dplit romsets, although to be honest the use of Dplit sets by users who don't understand how to use tools like RomCenter has been one of the biggest drains of energy for supporting MAME cores IMO. Understand that I'm not including you in that statement.

The advice in the RetroPie docs about romset format is derived from many many support threads on their forums: https://github.com/RetroPie/RetroPie-Setup/wiki/Validating,-Rebuilding,-and-Filtering-ROM-Collections#crash-course-in-arcade-rom-terminology

All that being said, in my younger years I myself added support for Split romsets to the other libretro MAME playlist databases. Those databases I made also include GAME_NOT_WORKING romsets in playlists. I've had some regrets about doing so just because I think it sets people up for confusion and failure when they don't understand Split sets and then delete a Parent. Or they don't realize that the Parent version of romset is GAME_NOT_WORKING but the clones are. Etc.

Have you considered using the standalone apps I wrote, RetroArch Playlist Buddy and Simple Arcade Multifilter?

Those are part of the tools I use to create these RetroArch databases. If you use my standalone apps, you have total control over using Split or Non-Merged sets and can control the inclusion of various categories as well as GAME_NOT_WORKING romsets. (The standalone apps also generate playlists that match the thumbnail repo.)

I hope this doesn't sound glib, but I do want developers and "power users" to be able to use the core how they want, while making it easier for new users to have a smooth experience with complete, working romsets.

Let's keep the conversation going if there's more to talk through! Otherwise I hope you'll keep in contact as you use these tools and the core.

[metchebe]

So, I have two observations for you to have in mind regarding keeping the split rom sets:

1. Someone with a large collection of split roms will have to rebuild them for the scanner to work (this is not my case, I have a very small collection). This would most likely be a "power user".
2. Other databases (FB Alpha for example) do pick up the split roms, so you will get a mish mash of parent roms in MAME 2003-Plus and their split clones in another playlist, which can be very confusing. For example, this happens now with simpsons and split clone simpsn2p which end up in different playlists.

Thanks for your time!

[markwkidd]

Thank you -- I'm specifically going to bear your feedback in mind as I make the next round of updates to the MAME 2003-Plus docs.

For what it's worth, I rebuilt a complete MAME 2003-Plus collection (including samples and CHDs) from Full Non-Merged to Merged as part of a discussion earlier this week about the possible value of adding Merged support to the core (Merged is not a priority but it does come up from time to time). The Full Non-Merged collection weights in at 31.96GB whereas the Merged collection totals 29.30GB.

That's a space savings of 2.66GB or 8.3%. I didn't do a Split set because I wanted to compare the two most extreme formats, but I'd educated-ly guess guess that a Split set saves about 7% from Non-Merged. I think that the person I was talking with -- who is used to Merged sets with current MAME -- was surprised that there isn't more space savings.

As I told them, these differences in size were more significant back in the days of low-speed internet and burning CD-R discs. And also these Split and Merged sets may still make more of a difference with current MAME's gigantic library of supported romsets. :) All this is just my food for thought hehe