WIP - Compatibility Table (dat magic)

[mahoneyt944]

I've decided it would be beneficial to create a compatibility table for our supported roms list since most are not reliable to say the least.

While this list will never be perfect. My goal is to create a table that can give us a general idea of where a rom stands in the core. In order to keep this list from becoming quickly outdated I want to pull this data from a reliable source, our very own DAT file. We can easily find out the driver status, color, sound, samples, and bios files from this source. Since this will be generated from the latest dat file each time it should be relatively up to date. This will also flush out incorrectly labeled roms hiding in our drivers with users feed back.

Eventually it would be nice to add a notes section for each rom and users can submit a PR to add these notes. Since these will be approved from the mame2003-plus team we can weed out bad notations and correctly label directives if required for a rom to be played. This will hopefully get the mame2003-plus user base more active with the development team.

This table will be available from this repository eventually. I quickly threw together a "proof of concept" example. I'll include these files here for anyone who wants to play with it.

The included html file is not accurate. This is just to show an example of what it might look like. The final product will be generated from a provided xml file.

Also included is a quick batch file I made to skim the xml dat. I won't be using this to make the final product but it was useful for generating quick lists from the dat.

compatibilty.zip

[mahoneyt944]

@arcadez2003 @markwkidd @Wilstorm

To add to this, I'd like to set up this table as automated as possible. My initial thought was to create a tool that would read in the dat file and spit out the table which then could be copied and pasted here.

But it would also be cool if the table was slightly interactive. Using check boxes to sort the table items. For example only show working roms or games with or without something like samples or bios files, 4way, 8way etc. Anything we can pull from the dat. We could also add clickable table items such as the samples to display the sample.wav names the core expects to find in the sample.zip

I'd like for this to be a go to source for compatibility with our core since this information isn't easily gathered without 3rd party help or outdated lists floating around the web. I'll probably just start with a basic table first then expand from there.

Any suggestions would be appreciated and if any of you have an interest in developing this with me I'd be glad to have you.

[meowthed]

this'll be a good idea, though another table in dedication to protection might be a consideration.

[mahoneyt944]

@arnoldsecret Right now we can include everything except imperfect graphics tags as they aren't covered in the dat https://github.com/libretro/mame2003-plus-libretro/blob/69c0a52c160d52cd82826527eebeadffa45e6163/src/info.c#L658-L688

[Wilstorm]

@mahoneyt944 - That would be great, I like the idea of pulling from the current DAT. There was a game I was going to ask you about, but I can’t think of it off the top of my head.

When I get in front of a computer tomorrow I’ll have to download the example file and look. Definitely nothing available like this around. All the lists are crap except for a few people that maintain them themselves.

[mahoneyt944]

@Wilstorm the example is just a mock-up, So it's not accurate. I just pushed a beta version of this program. It doesn't generate a chart just yet but it's a good start. I'll probably release the next version soon. 😴

[Wilstorm]

@mahoneyt944 - Ok that sounds good. Hey you wouldn’t happen to know Python by chance? I need some help extracting specific information from the DAT file to generate shaders on the raspberry pi. Using crt-pi and zfast. I think it would only be a dozen lines of code at most but I’m struggling a little bit with regex and the best way to tackle it. If I can get it working then I figured I could also use it for mame2003 and possibly FBNeo but I need to look at the FBNeo DAT layout.

[Wilstorm]

@mahoneyt944 - It looks real good here. I like the colors chosen. It's simple, clean and easy to read, it looks great to me.

[mahoneyt944]

Sorry I'm not a python guy. Maybe someday. I hear about it too often these days.

[Wilstorm]

@mahoneyt944 - No problem, I might open a new issue as not to muddy your thread and to show you what I've done so far. Here's the repo https://github.com/Wilstorm/pi_shader_configs. It is fully working right now but it's not completely accurate as games are added. Hence pulling info from the DAT. They are a night/day difference on even low end hardware like the Pi, they look incredible. Basically I need to pull these 7 fields:

game name="dkong" video screen="raster" orientation="vertical" width="240" height="256" aspectx="3" aspecty="4"

and write them out to a file like so: dkong,raster,vertical,240,256,3,4

[mahoneyt944]

You might want to skim through the version 2 beta of dat magic. It's very similar to what you're doing just with different info being pulled. Just pushed it.

[Wilstorm]

Yeah when you started this project I thought it's similar to what I've been trying to do. The Python script is long if you looked at it but it's mostly calculations to get the ideal dimensions for both vertical and horizontal games. Then it generates a custom cfg that you just dump in a folder and you're good to go. Even like the SNES and other console looks amazing with shaders.

So I looked at the new tool and see the datmagic.c, what do I do with it or how do I run it? I did COBOL, RPG, Fortran, etc. 30+ years ago so I can follow code and even updated that Python script quite a bit but I am by no means an expert. I mainly do firewalls, routers, switching, VMWare, some NetApp, etc. basically communication infrastructure or any public facing services type stuff for the State.

[Wilstorm]

@mahoneyt944 - The other thing I was going to mention is you can add a sampleof tag to the parent and it should work fine if it simplifies things. The only issue is ROM management tools will see it as 2 errors in the log if the samples are missing. The first error will be the samples to the parent missing and then the second error will be generated due to the sampleof tag if that makes sense. Basically it see it's own samples missing but it also sees itself as a clone sample set with missing samples.

[mahoneyt944]

I normally git clone the whole 2003+ project to my home directory then change into the datmagic directory with my rpi3. Then just type

gcc datmagic.c

To compile the code, then you can run it with

./a.out

[mahoneyt944]

I fixed the sample issue already in version 2 without modifying the dat.

[Wilstorm]

@mahoneyt944 - Nice, I'm just catching up now. My PC went kaput over the weekend and I spent the better part rebuilding it. I had Windows 7 and cut over to Windows 10. It took about an hour or so install the OS and drivers and then like a day and half more tweaking the environment and installing all the apps as it was before. What a pain! =/

[mahoneyt944]

Man that sucks. Glad it's up and going.

[mahoneyt944]

@Wilstorm once I finish up this program. I can modify it to log all the info you want from the dat. I just looked at it and it would be very easy to pull with some quick tweaks unless you want to tackle it yourself.

[Wilstorm]

@mahoneyt944 - That would be great! I am doing some testing right now as I was messing around today and I was able to get a working proof of concept in Python.

I did peek over at the FBNeo DAT and it just doesn't have the same information fields contained in mame2003 and mame2003-plus has or I would have loved to set up shaders with it too as that's the other core I use mostly besides mame2003-plus.

Here's a link of the RetroPie site that explains and has some nice screenshots of the shaders. https://retropie.org.uk/docs/Shaders-and-Smoothing/. I think shaders are a front-end feature so basically they are part of Retroarch.

Basically with the script you put in the core (mame2003-plus, etc.), the shader (crt-pi or zfast) and the target resolution (1920x1080) and creates a file basically like this:

# Auto-generated crt-pi-vertical.glslp .cfg
# Game Title : digdug , Width : 224, Height : 298, Aspect : 3:4
# Screen Width : 1920, Screen Height : 1080
# Place in /opt/retropie/configs/all/retroarch/config/MAME 2003-Plus/
video_shader_enable = "true"
video_shader = "/opt/retropie/configs/all/retroarch/shaders/crt-pi-vertical.glslp"
# To avoid horizontal rainbow artefacts, use integer scaling for the width
aspect_ratio_index = "23"
custom_viewport_width = "896"
custom_viewport_height = "1080"
custom_viewport_x = "512"
custom_viewport_y = "0"

I have a question for you but don't quite have the time tonight to write it up properly. It's nothing big but I wouldn't mind getting some feedback from the "guys" here and not the RP forums.

[mahoneyt944]

@Wilstorm I went ahead and modified my code for dat magic to spit out the video code you wanted. I ran it for 2003+ and 2003 and zipped them up for you. I included the modified version if you want to mess with it as well ( datmagic_mod.c ).

video.zip

[Wilstorm]

@mahoneyt944 - That's perfect! I've been trying to get that information current for several months so this is great!

I have a moral question for lack of a better way to describe it and was hoping for some feedback. The original Python script linked above was written by dankcushions. It worked but it was more like a brute force script that ran what I called a "99 loop". Literally it would loop the calculation over and over until it found the optimal settings. Then a guy named AndrewH came along and overhauled his entire calculation routine to make it much more efficient and accurate.

Here's where things kind of starting going off the rails. Retroarch changed its index from 22 to 23. This upset dankcushions and he refused to update the script until Retroarch came to some "solution" to not change the index in the future. He was aware this was literally changing 2 lines of code, of one number. It was the first time it had been changed since the scripts creation.

At the time I didn't know much about how Github worked so I took his script, made the changes, uploaded it on a free file service and linked it on the forums.

Later someone pointed out the curvature command line argument wasn't working. So I spent a few days studying the code. I'm still a rookie and what takes me two days someone with more experience could figure it out in minutes but I do get a sense of satisfaction and don't mind spending the time learning. I posted a message to dankcushions and said it was broke after this specific commit he had made. He still refused to update or fix the script. Again I made the changes and uploaded them to free service for others.

After that I just kept modifying the script with more cores and uploading them for others. I also added a new command line argument to use either crt-pi or the zfast shader but they were two scripts. They look slightly different but both look fantastic so it's a matter of taste. After taking some time I integrated the new command line argument and combined the scripts as one.

Dankcushions never said a word over the few years I was modifying and uploading the script for others. The links kept expiring after 30 days and requests would still come for the fixed code so I finally uploaded all the changes to my Github so others didn't have to ask all time and could get current changes.

I linked it on the forums and he was upset about that too and he all but said I "stole" his work. I said why in the hell would I do that and post it in your original thread. The weird thing is he had no issue with me linking updates through free services for years. So after that I set the repo to private for several months.

He informed me if I want to break down all my changes and submit each change as individual PR's so he could see each change. I could do that but he had no intentions and he was pretty much done with the script. I guess he wants someone else to maintain but he's all but done with the script.

I had no intentions of doing that. The scripts is currently broken on his repo and has been unchanged for 3 or 4 years now. I basically have a derivative works with many changes and now a current "database" of the mame2003 and mame2003-plus data to generate scripts. Your work inspired me to take another shot at Python. I figured eventually I could use Python to pull directly from the DAT into an array, process the data and generate the scripts.

I would like to open the script to others to use as they please. Do you see any issues with that? The one thing I did do wrong is I didn't credit him in the notes as the original author which I would do now. If you look here https://github.com/Wilstorm/pi_shader_configs/blob/master/pi_shader_configs.py it's basic code, it's not an organization, it's not actively been worked on in years and he stated he had no intentions of furthering the script, as it has served it's purpose,

I know it's going to really set a dark tone as he wants changes to be made to his code only but he doesn't want to do any work or update it himself. I wanted to get some objective feedback. If it seems like a bad idea I will hold or just open it and let others discover it on their own.

I do think there's a still a decent user base that likes to use the shaders this way because it corrects some issues with a "rainbow" effect on vertical games through the calculations of very specific resolutions at the expense of a slightly wider or narrower screen once it's been filled vertically.

Normally on an HD TV (1920x1080) the games would be 1440x1080 (4:3 ratio) or 810x1080 (3:4 ratio). Notice it's always 1080 (filled vertically) on the y axis. It's the x axis that changes. The script changes that ever so slightly to align with the shadow mask for a better shader experience and gets rid of the rainbow effect. To me it's that oil slick look of oil on water outside after a rain, this basically corrects that.

If the games is upscaled less than 3 times such as (320x240) multiplied by 3 on both the x and y axis it doesn't apply the shader. Games like popeye or rampage have resolutions like 640x480 and don't get upscaled enough. The shaders are just ok. You need about 3 real pixels for each shader pixel to make a decent shader but there only a few games with higher resolutions like that.

[mahoneyt944]

Well that's sounds like a can of worms. All I can say is that it's perfectly fine to create a fork of someone's project and add your own flare as long as you credit the original and aren't profiting from it of course.

[Wilstorm]

Yeah I don't think he has much coding experience. If you talk with him you'll swear he does and he'll often say something like I could probably fix that code but I have no interest if someone else wants to take a crack at it. He's not like you, arcadez, grant or mark that truly can program. He talks a big game and that bothers me. He does know a lot of things but programming isn't one of them. I don't mean that in a bad way but it's ok we aren't all on the same level of programming expertise.

It shouldn't be the way regardless. I feel very liberal about code, if it benefits others, share it. I did make the mistake not crediting and should have known better.

Ok I'm just going to push forward and hope for the best. I don't have bad intentions but I feel passionate about shaders. I feel they add so much visually. This is a good fit for me to tinker with and try to improve. I appreciate the updated data, it is fantastic! :)

[Wilstorm]

@mahoneyt944 - It's not much and could be done better but it works just fine to extract the data with Python. I just grabbed two fields for testing. I'm just going to use the files you generated as they are fully ready to go.

import re
f = open("mame_dataonly.txt", "w")

with open('mame2003-plus.xml', 'r') as fd:

    for line in fd:

        match = re.search(r'game name=("(.*?)")', line)

        if match:
            gname=match.group(1)[1:-1]
            print('game name = {}'.format(gname))
            f.write (gname)
            f.write (",")

        match = re.search(r'orientation=("(.*?)")', line)

        if match:
            orientation=match.group(1)[1:-1]
            print('orientation = {}'.format(orientation))
            f.write (orientation)
            f.write ("\n")

f.close()   

[Wilstorm]

At some point I would try and integrate with the main script to read from the DAT directly but it's not a hurry. I'm going to sort the files you created alphabetically and then upload them so the shaders cfgs generated are current.

[mahoneyt944]

I'm still undecided on a few things with it but at least dat magic generates a page now. Still in beta. More work to come. attached in zip is the first html page.

datmagic.zip

[Wilstorm]

It looks good to me. I have minor suggestions but definitely do what you want.

I would possibly remove the green since most are good running games and then you stop scrolling only when you see yellow or red OR I would add green to the sample and bios columns.

To me those last two columns just look like they are hanging in the breeze and don't belong to the set. There's color in the first 3 columns regardless of state and then nothing in the last two. Since samples and BIOS are probably all good and green.

Maybe a small legend at the top that shows what imperfect means (i.e. good = runs fine; imperfect = runs but may have issues; preliminary = ?). I don't know what that means.

Also I noticed 'dkongjo1' shows as bad dump in the description [in the DAT] but it shows good on all fields.

[Wilstorm]

By the way I do really like the BIOS field, I would have never thought to add it.

[mahoneyt944]

Yeah I could probably take the green out. The "<driver status" tag is where those 3 color fields are taken from, so it matches the dat in that sense. Bad dump would be its own field, and we also don't have imperfect graphics which is a big hitter we're missing. Then there's games that are listed at not working so preliminary shows in the driver status but the other flags aren't set so they show good anyhow.

This chart shows the flaws in our dat. Lol

[mahoneyt944]

here's a version without green and a brighter yellow

datmagic2.zip

[Wilstorm]

I like it better. Now I was able to quickly spot 'protection' that I had missed last time in the sea of green. Does protection have any meaning for the average user or just change it to good?

Well...since you brightened the yellow it seems to over power the red now. My eye is drawn to the 'sunny' yellow over the red. The red on my monitor looks more in the pink mauve range. I only say that as I don't color correct my monitors and I have two at home that are worlds apart. One is heavy in the darks and the other is heavy in the lights. When using Photoshop I think the slightly darker one is more correct.

So if I see 'preliminary' that is a show stopper bug that the game won't run? Some have preliminary graphics with imperfect sound or just preliminary graphics.

I think if the target demographic is the average user that doesn't want to dissect the DAT, even with the minor issues this is perfect and a huge step to allowing anyone to get a quick view of what works and what doesn't.

[mahoneyt944]

All these words stand for something different in each column.

Driver status

• preliminary means game not working
• protection means unemulated protection chip

Color

• preliminary means no color
• Imperfect means close but some colors wrong

Sound

• preliminary means no sound
• imperfect means missing some sounds.

Currently we don't have imperfect graphics listed because they aren't in the dat, but could be added then used in our table.

We also aren't checking fields like bad dump /no dump as I'm not sure how I'd want to display them just yet. These don't always tell the whole story. Sometimes a bad dump is just a borrowed chip to fill in where an actual hasn't been dumped. So the game would work still.

[Wilstorm]

Yeah it's a lot of information that's useful.

Maybe reserve the colors for "working" status.

Clear is working.

Yellow for working but with glitches.

Red for non-working, heavy graphical glitches or audio cues needed to complete a game or play it successfully. Basically games that are unplayable.

A small legend on the top with a color block for working (clear); working with glitches (yellow) and non-working (red). Then add a small note pointing to a more detailed legend at the bottom with all this other good information.

As a user if a game has a bad dump ROM but runs that's all I need to know really but for a coder they might want know there's a bad dump in a set. So mark the game as working (clear) but the field shows 'bad dump', 'imperfect' or 'protection' etc. for the coder.

Just throwing ideas out there. I know I have a spreadsheet of working games in different emulators but it's very busy with all the colors.

I remember back in the day bad dump ROMs were flagged using an inverted CRC (was used to show a 'clean' set in ClrMamePro that way), now I think they just straight use keyword 'bad dump'.

[Wilstorm]

I could see it going several ways depending on how you want to present the information really. It can be simple or detailed depending on your vision.

[mahoneyt944]

The information in the dat is just spread out and can be misleading. Right now dat magic is displaying the raw data correctly but that's not exactly what we want to see. We need to interpret the data and form a conclusion with it so the end user can understand what they are looking it without knowing the core's jargon. Things like the word "graphics" do not mean the same as "color" so where a game can be using the right color pallet, The graphics might be scrambled.... Or the graphics might be dead on accurate but the whole screen is only shades of blue or whatever. Color and graphics are separate. Unfortunately right now graphics are not listed in the dat. Maybe should be?

I'm trying to make the chart very simple yet informative. I'll think this over awhile and see if I can come up with something a bit more clear. And the colors can be adjusted as well that's easy. I'd like to pick colors that standout but don't over power the overall look of the table.

[Wilstorm]

@mahoneyt944 - Sounds good, yeah I agree. They sound like two distinctly different issues. I agree with the KISS principle when it comes to conveying information. Clean, simple & informative all are a win-win! :)

[mahoneyt944]

@Wilstorm OK... took it a little further. Added the graphics flag, played with the colors more, and added the proper descriptions. how's this look for you? I'm still working on the bad dump/ no dump fields.

datmagic.zip

[Wilstorm]

@mahoneyt944 - I like it, well done! I think you're close to rolling it out once you figure out bad/no dumps. You can give it your stamp of approval for production! :)

There's a good amount of information there but it's still easy to navigate. The colors are clearly distinct and none are overpowering. I like your new choices here.

The major issues seem to be highlighted in red, minor issues in yellow and green you're good to go. I was able to navigate around and easily figure out what games aren't working.

[mahoneyt944]

I like these colors too. What are your thoughts on a game name description?...might be easier for users to search for but does takes up more space.

Also, could you tryout the latest 2003+ dat in clrmamepro? I added the graphics field to it. Small addition, but I want to be sure it doesn't create errors or anything.

[Wilstorm]

@mahoneyt944 - That's a good point many will not know the ROM name vs. the game name. Shoot, I don't see a way around it. As an average user I would never think to search for puckman for PAC-MAN, etc. Some are straight up and would be easy like digdug but they may search for dig dug (with a space) and it wouldn't find it.

Yeah I will test it tonight. I have my main ROM setup at home so I'll run it through the paces and let you know later today.

[Wilstorm]

@mahoneyt944 - The set builds fine with the new graphics="good" tag. Also I don't if you want the SHA1 for the new additions headhonch and ooparts, they only showed the CRC32.

headonch:

    File: epr-16814.ic34
  CRC-32: d8dc6323
   SHA-1: e7e891324764641691dcb63e5222f2ed9207fb96

    File: epr-16810.ic4
  CRC-32: 90af7301
   SHA-1: 227227cb5d0df6612bac7b4c94b99e2287686ccd

    File: epr-16811.ic31
  CRC-32: 91f3b5f1
   SHA-1: 15cbe7a172dde7de7b73f0c9eeddfee41e8d1f80

    File: epr-16812.ic32
  CRC-32: 091cf538
   SHA-1: 04673678f543743b395edea39ad4ee6177436dc0

    File: epr-16813.ic33
  CRC-32: 3268e38b
   SHA-1: 10ded2be01465014ca9e6c64ffab1190ec985359

 ooparts:

 File: mpr-15616.ic34
  CRC-32: 7192ac29
   SHA-1: d3028a9bbb7faa733285cf7e47fd840ec0d0bf69

    File: epr-15613.ic31
  CRC-32: 35381899
   SHA-1: 524f6e1b1292542079589275e20f45c2eb68605c

    File: epr-15614.ic32
  CRC-32: 8dcf2940
   SHA-1: f72630e8a26e7f2089da56878a1599268c355246

    File: mpr-15615.ic33
  CRC-32: 42755dc2
   SHA-1: cd0aa79418b922266c5d41bf24b9136f9f105dc5

[mahoneyt944]

Yes I'll tag @arcadez2003 since he added these recently. Just to be sure about the missing sha1. @Wilstorm listed these in the previous post.

[arcadez2003]

@mahoneyt944 @Wilstorm The guru released those romsets people were expected to rename them and boot them in MAME via tantr, i added them as proper standalone romsets but TBH i didn't know where to get the SHA-1 values i'll update the romsets adding those in shortly

Regards

[mahoneyt944]

@arcadez2003 on my windows pc I can just right click on any file and there's an option for sha1 and crc. Not sure if that's standard or not but works for me. Just unzip it first then right click on the individual files.

[arcadez2003]

Aye i see that now i misssed the CRC / SHA option i was clicking on properties instead, well at least i'll know going forward how you obtain the complete rom data cheers

[Wilstorm]

@mahoneyt944 @arcadez2003 - I just setup the newest Windows 10 (20H2). I don't think Windows by default has any way to generate hash values unless the manufacturer or whatnot added it to their image.

I like to use HashCheck Shell Extension, a nice freebie on Github. It adds a tab to the properties menu named Checksum. You can can highlight one or more files, right click, properties, checksum and it will generate the several hashes per file like CRC32, SHA1, MD4, MD5, SHA256, etc. Then you can CTRL-A to grab them all or do the standard copy-n-paste with CTRL-C/CTRL-V.

They have a bunch of programs you can run and dump files into them that work but I prefer to just have it on the right click/context menu is super convenient.

@arcadez2003 - That's pretty cool to say Plus has some "exclusive" games!

[mahoneyt944]

@Wilstorm I'm not sure exactly. I never installed anything on my laptop to add it, knowingly at least. It came with the right click, sha1 crc option in the menu as far as I know.

[Wilstorm]

@mahoneyt944 - Yeah as long as I can remember, back to WFWG, Windows hasn't had any type of built-in 'hash generator' but then again I've seen weirder things I just can't explain. What's that saying ...

“To err is human but to really foul things up you need a computer.”

[mahoneyt944]

Im not real sure but see its here ha.

[Wilstorm]

@mahoneyt944 - Oh yeah I believed you. I see you're using Windows 10--Dark Theme! ;) Take a look at my screenshot. Mine is slightly different. I have Skype, WinRAR, Scan for threats (McAfee), Create Checksum File... etc. by default Windows 10 doesn't come with any of those. To me they are just Shell Extensions added by 3rd party programs to the Context Menu (right click menu). If it's a Dell, HP or whatever they might have added it to the image but if you did the install from scratch then I have no idea as it's nothing I've seen in Windows 10. We have deployed hundreds here at work.

I know you can use PowerShell and Get-FileHash c:\windows\explorer.exe -Algorithm SHA1| Format-List to generate just about any hash value type by changing "SHA1" to whatever you want to generate. Fairly clunky way to do it though.

I would be curious what program that one is as it's directly on the context menu which is nice. The one I use is under the Right Click | Properties | Checksum tab but it can do multiple files at once.

My context menu screenshot:

HashCheck Shell Extension screenshot:

[mahoneyt944]

Mines an HP, and it had windows on it already when I bought it new. So I'm not sure if it's 3rd party or not. I'll poke around later and see if I can figure it out. It is a nice feature though.