Possible missing DAT info

[Wilstorm]

@mahoneyt944 - This might have been caught as my DAT's are from January but I thought it's worth mentioning in case some people are deleting good ROMs. It seems that all the BIOS ROM sets are missing the <game name="neogeo"> tag. When I load the DAT into ClrMamePro it recognizes the name only but with not the individual ROM files.

I think it will basically delete the ROM files and create an empty zip named neogeo.zip. It's the same for mame2003.

Here's an example of the bad neogeo BIOS ROM entry: <game runnable="no" name="neogeo">

I think it needs to be similar to below. <game name="neogeo" sourcefile="neogeo.c" game runnable="no">

Newer DAT's do something like this: <game name="neogeo" sourcefile="neogeo.c" isbios="yes">

I don't know if the game runnable tag is used or not but I think ClrMamePro does leverage the isbios tag but having both wouldn't hurt anything. I find the extra tag info useful. I think the important piece here is it needs a game name tag to create the proper zip and contents.

[mahoneyt944]

I thought it typically added the bios roms to the gamerom.zip file when creating a romset? Is this not the case?

[mahoneyt944]

Also it's not a "game runnable" tag. It's just a "game" tag and "runnable" is just a listing under it. The order shouldn't really matter I wouldn't think.

[Wilstorm]

It depends on your settings. By default 'Separate BIOS sets' are off regardless of merged, non-merged or split-merge set creation. It's not a given they will be merged if that is what you're trying to force.

Right, the order is irrelevant but you have to look at it from ClrMamePro's point of view of how it tracks set relationships and not how the core sees and runs it. I don't want to run/delete anything without making backups but this is a snap of how ClrMamePro sees it. It doesn't show any ROMs for the BIOS sets.

It's kind of odd but within the Libretro/RetroPie community most use non-merged sets but outside the community everyone else uses split-merge mostly.

At a minimum it's an unorthodox way of doing it vs. the standard. Why use name vs game name? I know it's a BIOS set and un-runnable but it seems more prudent to stick with the standards because you have to use modern tools like ClrMamePro etc. to parse even old DAT's. By keeping to standards it allows them to work even though they are older than dirt sets.

I'm not sure why but modern DAT's even have complete "monitor" info like resolution (x and y), rotation (90 or 270), raster or vector, aspect ratio (x and y), etc. for BIOS ROM's. Basically complete info on it but I still use the tags like <isbios>, <isdevice>, <ismechanical>, etc. to keep or discard certain games.

Trust me I've looked at a lot of DAT's and mame2003-plus is by far and away the cleanest most complete DAT even though you seem you have some OCD for a super clean DAT and cut out things that make very little difference. I'm just giving you a little crap. The time to parse the DAT is negligible at best. I find the extra info quite useful as long as it's organized and appreciate all the updates. I'm looking forward to trying the sub game arcadez added samples for, there's never enough sub games. ;)

[Wilstorm]

If I could add one other thing is having the sourcefile="" is really helpful for tools like DatUtil if you've heard of it. You can pull all of the NeoGeo games and clones for example keying off that tag or other tags. I know Grant had added it (sourcefile) to the DAT at one point as it was as close to 1G1R as we were going to get. Now that it's removed again several ways of paring down sets is crippled again. I prefer the full and complete header info over the non-standard truncated versions. I know for the typical user it seems irrelevant but once you starting using tools people have created for that extra info specifically it's really useful.

[mahoneyt944]

I haven't cut anything out of the dat? There was only the one time the sample tag got messed up and that was code ported from grants repo. Are you sure your using the newest commit?

[Wilstorm]

@mahoneyt944 - I'm not using the newest commit. I'm behind for sure. Maybe Grant made it special when asked and not part of the regular code. I tried to make a case for the info being useful but it is what it is. I'll close this out and thanks for looking!

[mahoneyt944]

Our newest commit includes the "name=", "sourcefile=", "cloneof=", "romof=", and "sampleof=" when available in the game tag. The biossets are probably handled differently so I'd have to look at those.

[Wilstorm]

@mahoneyt944 - It’s not a big deal. I think Grant made me a few DAT’s with extra info through these past few years. It’s all good. I think the core has one of the best DAT’s regardless.

I wrote a Python script that parses all the Libreto core DAT’s to create the info similar to what you did for mame2003 and Plus. The newest DAT’s are insanely huge like around 500,000 lines.

I also took a crack at FBNeo that has no XML info at all in the DAT. I had to parse all the code and drivers. It takes about an hour to run. I think it’s a bit slow as it uses regex matching to search for the data. Regex can get ugly quick. The drivers have spaces and tabs intermixed, comments on the end, different driver layouts, “foreign” characters that make Python balk. We use something like the CP2132 character set here. Or I think it’s tied to your keyboard character set loaded. I need to do a bit more testing to verify the integrity of the data. I learned a bit more about Python than I cared to know but it was a fun learning experience, I think! =\

[mahoneyt944]

Datmagic is still in our tools section of this project. It could be tweaked to pull anything from the dat itself. Right now I have it integrated to build mame2003, mame2003-plus, mame2010, mame2015. But haven't done much with it since.

[Wilstorm]

That looks pretty neat. I love the printf "DAT MAGIC" at the beginning. :)

When I mentioned mame2003 being so tidy and easy to work with. Basically all you have to do is pull the fields and write them back out as a comma delimited file in whatever order you need.

In my script I trigger the start of a new game something like this:

gameName = re.search(r'game name=("(.*?)")', line)

and then do a boolean check:

if gameName: <- (returns 'true' if found)

You have to read a bunch of lines to find all the info needed to make calculations. I initialize a list and add them by subscript. So you might see something like rotate="270" in mame2015. So that tells me the orientation="vertical", aspectx="3" and aspecty="4". Also with vertical games you have to invert the resolution. So width="304" and height="224" becomes width="224" and height="304".

After I gather all the info I break each step into a function like so for example calculating the aspect ratio: You have to convert the string to an int, perform the calculations and the convert it back to a string before writing it out.

def write_non_std_ar_games(count, gameFile, fileName, game):
    if fileName != "mame2010": # mame2010 has no 'monitor type' (can't calculate AR)
        if game[-1][7] == "rscreen":
            if len(game) == 2:
                game[0][5] = str(int(game[0][5]) * 2)
            elif len(game) == 3:
                game[0][5] = str(int(game[0][5]) * 3)
        elif game[-1][7] == "bottom": # Aligned vertically
            if len(game) == 2:
                game[0][6] = str(int(game[0][6]) * 2)
        if (game[0][5] != "3" and game[0][5] != "4") or (game[0][6] != "3" and game[0][6] != "4"):
            #print ('ROM = {: <10} \t Aspect Ratio = {: >2}:{: <2}' .format(game[0][0], game[0][5], game[0][6]))
            gameFile.write(','.join(game[0][:-1]) + "\n")
            count += 1
    return count

With mame2003/Plus you don't have to do any of these calculations. They are all done for you which is nice except the BIOS files are a bit messy.

ALL other DAT's use game name and also identify it as isbios or isdevice or ismechanical. So you have to get creative with BIOS files in mame2003/Plus without the header info.

I actually use game name to trigger the start of a new game and reset my counters, reinitialize my lists, cleanup, etc.

I also track and write out all the "other" types of "games". I write them to file and print a summary to screen like so:

Summary information for mame2016:
---------------------------------

Vector games (vector games don't use shaders) = 100
Games with multi-monitor setups = 312
Games with non-standard aspect ratios = 130
Games with no game information = 3763
Games tagged with 'isbios' (removed) = 69
Games tagged with 'isdevice' (removed) = 3044
Games tagged with 'ismechanical' (removed) = 15013

Total games with complete information = 14560

Anyway that was long. I've never done or much less looked at Python so it's my first attempt this past month or so if it's messy and not modular.

If you get a minute the whole point was to make shaders for RetroPie but it could easily be adapted for any platform by changing the path. https://github.com/Wilstorm/retropie_shader_configs

There are only few files so it shouldn't take but a minute. The main script, a batch file and two resolution DB's (mame2003 and mame2003-plus) for now.

The README.md explains it pretty well and also has a link to the original RP thread and some other good info that shows you in a few screenshots how aligning the mask on vertical games much improves the visual experience vs. a oil slick look and also horizontal to a lesser degree.