MAME to .neo conversion tool link

[carmiker]

The link in the README is to an archived repo with a "no maintenance intended" flag: https://github.com/city41/neosdconv

I have actually had better luck with this tool anyway: https://github.com/bodgit/terraonion

Maybe it's worth changing the link or adding this as an option? I'm also considering writing my own tool which I will maintain...

[sorgelig]

both tools are generally bad. They cannot be compiled to executable file. Terraonion version cannot be executed as go already changed too much.

[carmiker]

I've heard tell of a modern full MAME set that has been converted to .neo recently and has been made available at a popular internet archiving site.

[sorgelig]

the problem is not to find already converted roms but tool to make such conversion

[carmiker]

The person who created the above mentioned full set has created a tool in portable C but not released it publicly yet. I'll post that link here if/when that happens.

[Toryalai1]

The NeoBuilder Tool is available on Stone Age Gamer website: https://stoneagegamer.com/neosd-downloads.html#neobuilder

[Koston-0xDEADBEEF]

both tools are generally bad. They cannot be compiled to executable file. Terraonion version cannot be executed as go already changed too much.

Terraonion tool compiles fine into an executable. The code also looks reasonably clean at a glance.

A functional MAME to .neo conversion tool is actually trivial to write - just a couple of lines of shellscript does the trick already - but this excludes breaking encryption on a handful of protected ROMs. MAME does this on the fly, Terraonion neosd re-implements the MAME decryption algorithms and vast majority of the codebase is just decryption.

I can think of a few reasons, why re-implement all of it again in C:

• Enable encrypted MAME ROM loading directly in MiSTer
• Extend the .neo format by utilising empty space in the header. Bizarrely, Terraonion specification specifies 4K header, but it only reserves 33 bytes for a game name which cuts many names short. Besides long name, there's a whole slew of metadata which could be included in the header without breaking backwards compatibility.

So - not very good reasons.

[sorgelig]

Terraonion tool compiles fine into an executable. The code also looks reasonably clean at a glance.

Compiles fine for those who familiar with that weird language and not less weird build system. General user needs ready to use utility instead of spending hours or days to figure out how to compile it when project is no longer compatible with current version of compiler. Old good C/C++ is more portable than these weird modern languages.

[Koston-0xDEADBEEF]

I'm not very familiar with Go either, but it's a compiled language and this neosd tool seems to be competently written. Also BSD licensed, allowing binary distribution. I'll take a closer look and see how it could be leveraged. I'd prefer a C implementation personally, but that would be significantly more work without good enough reasons.

[carmiker]

Yes, the tool written in Go is our easiest way to go about this. It will convert most of the MAME set without a problem, and there are only around 30 titles that won't convert to something usable. Here is a (possibly incomplete) list:

burningfpb
fatfursp
gpilotsp
irrmaze
kf10thep
kf2k2mp
kf2k2mp2
kf2k3bla
kf2k3pl
kf2k5uni
kof10th
kof2002b
kof2k4se
kog
ms5plus
mslug2t
mslug5b
pbobblenb
roboarmya
samsho5b
svcboot
svcplus
svcplusa
svcsplus
viewpoinp
vliner
vliner53
vliner54
vliner6e
vliner7e

That being said, the person who created the "LunaGarlic" set has written a tool in C, but my understanding is that the code is quite messy because so many edge cases must be handled. I can see about encouraging this to be cleaned up and released.

Let's avoid extending the .NEO format. The supposedly unused area of the header is actually being used unofficially by the NeobuilderUI tool to record information about parent MAME sets already, and who knows what TerraOnion may have in store for the future. Yes, the format is imperfect, but we also have the first viable path away from unwieldy MAME sets, and even more fragmentation would be counterproductive.

[Koston-0xDEADBEEF]

Yes, the tool written in Go is our easiest way to go about this. It will convert most of the MAME set without a problem, and there are only around 30 titles that won't convert to something usable. Here is a (possibly incomplete) list:

Thanks! I'll take a look, most likely problem is just some missing metadata. For instance, mslug2t is exactly identical to mslug2 for all intents and purposes - only the program ROM has different checksum (due to small bugfix in game engine).

Let's avoid extending the .NEO format. The supposedly unused area of the header is actually being used unofficially by the NeobuilderUI tool to record information about parent MAME sets already, and who knows what TerraOnion may have in store for the future. Yes, the format is imperfect, but we also have the first viable path away from unwieldy MAME sets, and even more fragmentation would be counterproductive.

I agree that extending the format is a bad idea, because it is essentially a closed source format and even the documented part already contains proprietary data. Going down this path would mean forking the format into a MiSTer-specific one - which is also a bad plan.

Let's take a step back and assess the situation.

Benefit of .NEO format is self-contained single-file format - one file always contains everything needed to run. No need for external metadata or parent ROMs. It's also compatible with NeoSD. But, neither of these are strictly true. We are already diverged from the official .NEO format, which does not include decrypting anything, making interoperability impossible to guarantee. The official header format specification is also incomplete, some games require undocumented data rearranging.

Doing the conversion reliably also requires having much of same metadata available from MAME as is needed to load MAME ROMs directly. Huge benefit here is, that neosd already does all this.

Final benefit is being able to freely rename the .NEO files to have more meaningful names than the original, eight character limited MAME names. I personally am very happy about this, and it can be used to circumvent the bizarre name length limitation in .NEO file header.

...

Technically the best approach would really be to just support MAME ROMs as-is. This would mean having to pull all ROM metadata from MAME and re-implement all decryption/descrambling algorithms in C. This is more work than I have time for, but should be kept in mind in case opportunity presents itself.

So, for the time being, I will look into neosd to address known issues, document it for MiSTer purposes and compile a binary to run directly on MiSTer along with a simple script to automate a set conversion.

[carmiker]

I think the .NEO format implies the data is decrypted, but as you say, there is nothing in the spec stating this outright. When I started writing my emulator (Geolith) I was using the official NeobuilderUI tool to convert the encrypted romsets so that I could emulate them without having to think about whether any problems were due to decryption, or due to my emulation being imperfect.

For supporting the MAME sets, the other thing that must be mentioned is that the sets are subject to change. In practice this does not happen often, but over time the changes can build up and suddenly a good part of the library does not work unless you were diligent enough to keep track of changes -- and then your users also need to update their sets. The NeobuilderUI tool is behind already, and can no longer convert some titles in the MAME set. This is, in my opinion, the main thing the .NEO format solves. Then, there is also the issue of homebrew. If a homebrew developer wants to create a game for the Neo Geo, it is just much easier for them to use a format that should just work with any implementation supporting it without any further intervention.

I'm glad to see you're willing to put time into this, @Koston-0xDEADBEEF. I recall there were some headaches converting ms5plus and kof10th, so if you have problems with these or any other titles, I may be able to offer some advice.

For undocumented data manipulation, are you referring to swapping the first and second megabyte of PROM when PROM is 2MB, later games having the FIX data at the end of CROM, both, or neither?

[Koston-0xDEADBEEF]

I think the .NEO format implies the data is decrypted, but as you say, there is nothing in the spec stating this outright.

You're right - I was misremembering what I'd seen earlier in darksoft-to-neosd tool docs, saying NeoValidator fails on some converted ROMs:

KOF 98, Fatal Fury 2, Sidekicks and Metal Slug X are patched
to deactivate the copy protections in the darksoft build

Many more games are encrypted, but this lot has some additional copy protection mechanisms. I assume removing these is also required for use on the MiSTer.

For supporting the MAME sets, the other thing that must be mentioned is that the sets are subject to change. In practice this does not happen often, but over time the changes can build up and suddenly a good part of the library does not work unless you were diligent enough to keep track of changes -- and then your users also need to update their sets. The NeobuilderUI tool is behind already, and can no longer convert some titles in the MAME set. This is, in my opinion, the main thing the .NEO format solves.

This seems odd, considering the MAME sets are supposed to be just 1:1 dumps of the original ROM chips. A large part of a library changing sounds quite alarming, thanks for bringing this to my attention! I'd preferred MAME ROMs precisely because they (supposedly) have the same goal as MiSTer; being as accurate to original as possible, so in theory, following their changes would be the correct path. But if they've gone the way of Linux and are breaking backwards compatibility just because they can, it's a shitty path.

For undocumented data manipulation, are you referring to swapping the first and second megabyte of PROM when PROM is 2MB, later games having the FIX data at the end of CROM, both, or neither?

I was only aware of the former, in general the specification is just not very good. But, it's the best we've got.

[carmiker]

As far as I know, the only patching done by NeobuilderUI is on "sbp" aka "Super Bubble Pop". I think Darksoft just patched away the protection mechanisms for convenience. KOF 98 uses a very simple mechanism which simply replaces the values returned by reads to a specific address if certain registers are written to. Metal Slug X uses a more exotic scheme which I don't think has been fully documented, but we do have a reliable way to deal with it without patching. Fatal Fury 2 and Super Sidekicks use PRO-CT0 which MAME has emulated at a low level, so at least this is possible. High level solutions also exist (I use one.. for now). I've documented all of the protection mechanisms as verbosely as I could here: https://gitlab.com/jgemu/geolith/-/blob/master/src/geo_m68k.c?ref_type=heads

At this point in time, the likelihood of the MAME set changing significantly is low. Most of the time the changes are just filenames inside the .zip being renamed to match the silkscreen label on a chip when new information is discovered, or to change .bin to .c1 or .v1 for consistency's sake. Occasionally things will be redumped if it is discovered there was a bad dump. That being said, things are still subject to change.

The MAME format is indeed the best format for archival purposes. However, in my opinion it doesn't make a difference whether the decryption portion happens inside the emulator or outside it if it is still operating on decrypted data. To my knowledge, nobody has emulated the decryption in real time. I am pedantic enough to do such a thing, but I don't have enough time or information to make that happen at the moment.

[Koston-0xDEADBEEF]

Oh wow, that's very useful information, thank you!

Your code is very clean and pleasant to read.

[Koston-0xDEADBEEF]

Status update..!

As it turns out, @sorgelig's earlier comment "..spending hours or days to figure out how to compile it.." was a bit uncomfortably too on the nose, since it ended up indeed taking several days to find at least somewhat sensible way to compile the neosd tool for MiSTer. It's still not quite there yet, but this is a good time to at least make a record of things learned so far.

The neosd tool has a dependency to zlib (C library), and it turns out building a Go project with C dependencies can be troublesome. Trying to cross-compile was worse; out of three hosts tested (macos, freebsd and linux), precisely none provided all necessary toolchain components to target linux/arm (probably some Linux distro has, but mine did not).

Instead, I found there is a github actions provider using Qemu for various ARM targets, so I took to Qemu to first perform this locally. Getting the MiSTer linux kernel booted was unsuccessful, but Debian official bullseye raspi image did boot. Qemu command line below; kernel image, device tree file and initrd are taken from the image's /boot.

args=(
    -drive file=20230102_raspi_2_bullseye.img,format=raw
    -machine raspi2b
    -serial telnet::4444,server
    -display none
    -usb
    -device usb-net,netdev=net0
    -netdev user,id=net0,hostfwd=tcp::8022-:22
    -initrd initrd.img-5.10.0-20-armmp
    -kernel vmlinuz-5.10.0-20-armmp
    -dtb bcm2836-rpi-2-b.dtb
    -append "console=ttyAMA0 root=/dev/mmcblk0p2 rw rootwait panic=1 net.ifnames=0 cma=64M"
)

qemu-system-arm $args

This works and can even build neosd, but.. it segfaults on MiSTer. Oh, well - I got the build steps from it, and they are embarrassingly trivial.

This commit successfully builds an arm binary using github actions: https://github.com/Koston-0xDEADBEEF/terraonion/commit/3f6c3bc588036a0dc9566ff68387c68278f709b9

Only problem is, it uses Debian Bookworm as a base, and as such, the produced binary has an unfulfilled GLIBC_2.34 dependency, preventing it from running on MiSTer. I'm currently experimenting with older bases in an effort to find something that would output an actually usable binary, but besides the ARM runner itself being somewhat buggy, I am hopelessly clueless about details and differences of ARM platforms and Linux kernel features for it. So, it's essentially a crapshoot.

One final alternative I've yet to explore is modifying the Bookworm build so that it modifies the resulting ELF binary and just removes the unfulfilled GLIBC version dependency.

In any case, I'm starting to run out of energy (and time) to spend on this, so it might be a while longer until there's anything usable. Before then, there is at least a working github actions already for producing a linux/amd64 binary, hopefully doing the same with macos/amd64, macos/aarch64 and windows/amd64 won't be terribly much harder.

[Koston-0xDEADBEEF]

Alright; commit https://github.com/Koston-0xDEADBEEF/terraonion/commit/08400b57169be0468aebcb2fee07370beb4295ab produces an intact linux/arm binary that runs on MiSTer; but still same segfault as before. It looks like it happens even on the same linux image it was built on, so on the bright side at least I can next move from builds to debugging..

[Koston-0xDEADBEEF]

Filed https://github.com/bodgit/terraonion/issues/7 for now.

Let's see if this goes anywhere. I don't know Go at all, and I'd rather avoid learning why they're failing to get rid of ancient C requirement of byte-aligning structs. Portable code, but only on a single platform? Sigh.