Add support for MAME Software lists (primarily for CD based consoles)

Mame Software lists for consoles seem like they could largely be compatible with Retool, particularly for the consoles retool already has clone lists for. This would make it easy to have cleaned up CHD based dats, which are becoming more popular for disc media based platforms but aren't really documented by other DAT groups.

The software lists use parent/clone relationships already, and the description tag seems to largely match the redump/nointro naming conventions, note the ROM name uses a mame naming convention, but that seems to align with the way Retool tries to create a simple name to group parents and clones.

This would indirectly provide partial TOSEC support, as a lot of the CHDs in the SLs are sourced from TOSEC - the dreamcast for example list seems to lean heavily toward TOSEC, despite being a TOSEC sourced rip, the SL description field still appears to use redump naming conventions.

Here are some of the more relevant software lists: Dreamcast PSX Sega CD Neogeo CD Saturn

I did play around with the code to see how viable this is, current behaviour is the Software list gets rejected as an invalid DAT, it didn't take a ton of work to get some basic DAT parsing completed with region support. I got a bit stuck on successfully getting it to parse parent/clone to actually give any useful output, but I didn't want to spend much time on this if the new release isn't far off.

I appreciate the CHD focus, that's definitely where things are going for some platforms. There's even a good argument that MAME verified CHDs will see higher distribution, so the associated tools should support them.

There are a few blockers for this, however:

Most importantly, time. Supporting this involves a lot of work on new clone lists and logic, even considering Retool's existing clone list support and the established relationships defined in the MAME XML files. There are a lot of software list systems, which means a lot of testing for inconsistencies and quirks while making sure existing functionality isn't impacted. This isn't something I can justify the time on, let alone on an ongoing basis when MAME updates -- Redump and No-Intro already keep me busy.
The naming syntax in MAME SL description fields is inconsistent, which requires extra logic to overcome. A few examples:
- Regions are inconsistent: Euro and Europe are in the Dreamcast XML, as are Jpn and Japan. In other XML files region tags are outright missing.
- There are combined tags like (USA, alt) and (USA, prototype, 19940928), which makes parsing harder.
- There are split tags like After Burner (USA) ~ After Burner Complete (Japan) for dual name titles, which has implications for group and short names among other things.
Community engagement isn't high enough to fill the gap and provide clone lists for a meaningful number of systems to make MAME SL support viable. I'd rather Retool doesn't support MAME than do a bad job at it. This is also the fundamental issue with supporting TOSEC, although its rigorous naming system would at least make the logic part easier.
If I enable MAME SL compatibility, it's going to increase requests for arcade 1G1R, which just isn't feasible.

Regarding Clone Lists/Logic I understand your concern, that was why I suggested this be limited to the CD based consoles already covered by Retool/Redump/Nointro, at least to start - that could be done explicitly during DAT validation, and extended only if/when key issues are resolved for a particular platform. It's specifically CD based platforms that have the most demand as users struggle with managing lots of bin/cue/gdi files with emulators, chd has reached a point of near universal support, while zip/7zip archives are more haphazard and problematic - that's all I am personally interested in this function for. I'd be happy to help contribute with some of the logic changes/testing as well.

Regarding the MAME SL description field challenges, understood - will try taking that upstream to see if the MAME/MESS team will take description updates to match nointro/redump standards (or any standards you suggest). It turns out the current software lists covering cd media need nearly every entry to be touched and updated to be compatible with the latest versions of MAME/chdman anyway - see mame #2517. That said this bug could be considered another show stopper, at least until someone comprehensively tries to update the SLs. I've just converted a couple hundred psx/dc titles from redump/tosec and discovered this the hard way, need to explore how much work it will be to update the SL DATs and see how the submissions go, but it's just a drop in the bucket compared to the total number of titles that need to be updated.

Regarding the last two bullets, this is something that could potentially be disabled by default and left as an experimental function, at least until it can be seen whether some of these other challenges resolve themselves. Agree that expanding to full SL would need considerable community support, but leaving it as something that needs to be enabled by changing a bit of code would limit access to more technically minded capable of contributing.

I don't think any of these changes get close to MAME arcade aside from possible perception challenges of the MAME name - the software lists are fundamentally trying to do the same thing as nointro/redump, whereas the arcade xml is radically different, trying to describe hardware, dip switches, etc - I totally get staying away from that, and most front-ends have solved the arcade 1G1R problem in other ways anyway.

Sounds like the core issue isn't MAME SL support, but the converted images problem. RVZ has the same issue, among other formats.

Ideally DATs should support multiple image types to circumvent all of this, but that's a whole extra discussion that requires a lot of coordination between various groups and ROM manager authors. I started the discussion about a new DAT format some time ago, but it's not an easy proposition to get off the ground given the entrenched state of some of the systems involved. A less optimal solution would be a ROM manager that does the conversion back to ISO the first time, then caches the results for future runs providing the CHD hash hasn't changed.

With that in mind, you might find verifydump dovetails nicely with your CHD interests. It's probably quite a disk thrasher though as it doesn't look like there's results caching yet.

Eventually Nkit 2 looks like it is also poised to solve some of these pains, although it's an ambitious tool so I'm unsure how long it'll take to get there.

Agree it's not the SL themselves in terms of dat structure, etc - it's just highlighting a general lack of maintenance on these lists which make them less relevant in their current state, but the fact that it's the MAME project tracking these still make them a good source for retool to potentially support.

The issue in this case is tied to MAME versions and the SL - majority of the CHD hashes in the SLs were created by an older version of CHDMAN prior to a bug fix in 2016 that updated CHD metadata (and in turn the chd hash), and a few older platforms were hashed by an even earlier version of chdman which had more significant bugs which essentially corrupted the data. So the problem is actually the opposite of the one you are thinking of with verifydump - I had the original redump images - in this case I converted ~600 psx titles from validated redump images to chd using the 0.251 chdman, and only 7 of those actually matched the 0.251 SLs, primarily because those 7 were the only entries updated since 2016 and also match the current redump dats.

So right now the only way to get a CHD matching the header is to go and download the current SL chds with bad hashes directly, or to install a pre 2016 version of chdman to convert the dumps to use intentionally bad hashes - but even in that case there will be rips that won't match in cases where redump has switched to a new preferred version since the last SL update.

The silver lining here is that the MAME team will likely be more open to more wholesale updates of these dats due to this bug and the infrequent updates. Since the SL dat headers generally indicate redump is the preferred source, I'm thinking about ways to automate SL DAT updates - I don't see why matching the description to redump would be a major issue to the MAME team and it would make it easier to programmatically update the SLs to the latest preferred redump titles where possible.

Agree with you on the other points, would be good to see solutions for rvz, cso, etc - one challenge though is a lot of the teams creating these new formats only care about emulation, not about archiving/cataloguing - the dolphin team had no interest in adding a hash to the RVZ format header - see here, and the problem there is that the RVZ itself can change it's hash based on the compression setting used - no way to create a 'standard' hash for DATs unfortunately. CSO seems to suffer from similar problems of variable compression and no full original hash in the header, and cso has other performance issues anyway similar to zip/7zip on low end devices.

So that still leaves CHD as the front runner at the moment for compressed formats which are archive/catalogue AND emulator friendly. That said the SL lists don't catalogue these newer platforms yet where this would be relevant, not sure if they ever will or that requires the MAME team to take a stab at platform emulation for the newer platforms.

I'm going to close this one down -- I don't have any interest in supporting MAME software lists at this point in time.

CHD support is meant to be up next for development in Nkit 2, and I believe it's the top requested feature for RomVault. These are more appropriate channels for this sort of support.

unexpectedpanda / retool

Add support for MAME Software lists (primarily for CD based consoles) #233