Add a switch to remake ips patches to 'no-header' versions.

[i30817]

If a patch doesn't change the header, it's possible to turn a ips patch for a headered rom into a patch to a non-headered rom.

I normally use ipsbehead for this, but it seems very unknown, and it has the limitation that you have to know ahead of type the size of the header (in short, it applies to SNES and PC-engine ips patches that were supposed to apply to ROMs that have the 512 bytes header).

So there are a few opportunities here. If it's possible to validate a ips patch needs a header. I'm not sure that is possible, even passing the rom, with a header or not.

Otherwise the operation might be 'destructive' if applied to a patch that already is to a no-header ROM. Anyway, even destructive, i think it's a switch worth supporting, with a few warnings if the patch can't be ascertained to apply to a header version because ipsbehead is very unknown and it's sort of in the wheelhouse of this program.

I don't know if there are other consoles where the ROM dumps have header and non-header versions that could apply. I know that redump xmls have these { '.nes' : 16, '.fds' : 16, '.lnx' : 64, '.a78' : 128 } as the number of bytes to skip for the roms that 'require' a header to check checksums* (note, SNES sfc is not there because redump only distributes no-header versions because it doesn't 'require' a header). So basically i know those 4 don't count because even no-intro gave up on not requiring headers on them.

* of note, this is completely idiotic because nearly all the patches to nes on romhacking.net for instance, say they apply to 'crc something' and the crc something was lifted from redump and .... the patch also counts the header and applies to a completely different rom than the redump dump because the header is different. Great fucking job redump. Or maybe the fault is with the patch uploaders.

[Alcaro]

IPS is a very simple format; it simply contains a list of 'go to address 0x123, write bytes AA BB CC, go to address 0x1234, write bytes BA AD F0 0D' and absolutely nothing else. There is no obvious way to detect the header size - if there was, ipsbehead would do that.

Even with such a primitive format, I can think of three ways to guess if an IPS is for a headered ROM, but they're all fairly limited. The first is check the address of the last byte written. If it's 0xFFFFF, not headered; if it's 0x1001FF, headered. This works only if the target ROM is bigger than the source; if the last written byte is 0xFFDE3, that tells you nothing. The second is compare to another patch for the same source ROM. If one has several writes to addresses 512 bytes ahead of the other, that one's headered and the other isn't. But this requires having another patch, and also requires that both target ROMs were hacked with tools that edit the same parts of the ROMs. The third is apply the patch to both candidate ROMs, run in emulator, and see which has more plausible output. This, of course, requires defining 'plausible' in a computer readable way, which is far from easy. And if the patch only swaps some levels, you'll need to either teach the emulator to get past the title screen, or return inconclusive.

Unlike IPS, the BPS format contains size and checksum of the source and target ROMs. The format is, however, not header aware; it's all on Flips' side. If it tries to apply a BPS patch to wrong source ROM, it checks if dropping the first 512 bytes creates the right source; if so, it does, then copies those bytes to the target. The actual patches are perfectly normal patches between unheadered ROMs.

SNES ROMs contain an internal data block that tells what mapper/chips/sram amount/etc are in use, as part of the actual cartridge. NES ROMs do not, and require storing that information elsewhere, which is the 16 byte iNES header. I know nothing about Lynx and Atari 7800, but I'd guess they're same as NES.

As for the actual issue, I'm skeptical that'd be a good idea. As you said, IPS is a quite limited format; to me, ipsbehead existing is an argument to drop IPS and switch to BPS, not an argument to behead more patches.

[i30817]

You can drop ips creation, as long as you don't drop ips patching. Because countless patches are in ips and will remain there because people stopped updating them.

[Alcaro]

If removing IPS creation is fine, then the IPS format is deprecated, and adding other IPS functionality makes no sense.

Yes, IPSes being applied to misheadered ROMs is a problem, but I don't think Flips is the best way to solve it, and I don't think header size options is the best solution available within Flips (it'd have quite poor user experience, you'd have to guess possible header sizes until one works).

The trial and error part can be removed by adding a source ROM checksum option, which will make Flips try to remove various header sizes until it matches. But that still requires copying hex strings from RHDN. And in both this case and the above, and I have no idea how the graphical UI would look.

A more convenient experience can be created by making an RHDN integration tool as a wrapper around Flips, which automatically collects the applicable checksum. But then why not put the entire header removal function in the wrapper?

Or even better, solve the problem at its source by swapping the IPSes at RHDN. While many patch authors have lost interest in ROM hacking or otherwise won't update anything, the RHDN staff still exists, and could swap out the ZIPs if they want to.

(This is admittedly is a pretty big if. It requires not only approving the idea of hosting a ZIP the author didn't upload, but also verifying that every patch works on its respective source ROM, and updating every description and readme that mentions IPS or any IPS patcher.)

[i30817]

It's not only the backwards compatibility. A great many rompatch addendum depend on being not checked by checksum, because they have optional parts that have combinatorics numbers or say 'oh we're compatible with x,y and z if we go last' (or sometimes before).

Frankly, although i'm sympathetic, i doubt you'll ever get rid of people using ips, or wanting to use it, for many reasons.

At least with checksums 'like they're practiced' now. A checksum that checked just the bytes the patch will alter, and their position, maybe, although that would be a 'half full' half measure. Doesn't eliminate user error, keeps some flexibility, but users will not care.

[Europia79]

This entire "issue" could easily be solved if RHDN simply required Authors to upload their actual, full romhack instead of patches: Then, their server could easily generate a patch against the players specific ROM, whether that ROM is headered or unheadered, or even, whether it's (Rev 0) or (Rev 1) would not matter.

@i30817 ...Alternatively, if you desire to remake patches without the 512 byte header, then this could easily be automated via script (shell/bash/batch,etc) since Flips has a Command-Line Interface:

Example:

for file in Mario\ 64\ ~*; do flips --create --bps-delta 'Mario 64 - Super Mario 64 (#Nintendo)(USA)(En).z64` "$file" "${file/.z64/.bps}"; done

You would just have to go thru the grueling manual process of applying patches & testing the functionality in the emulator in order to verify that they were applied correctly and determine whether or not the patch info & instructions concerning headers was actually CORRECT.

You should be able to upload these patch corrections & fixes (if you want to do that).

flips --help has more info if you wanna pursue this route of CLI & scripting automation.

But yeah, SNES & PC-Engine do not require a 512 byte header: As @Alcaro said, SNES already has an internal header to store ROM size, Checksum, & other ROM info.

Only NES actually requires the extra Header information (because of the different Mappers).

For this (NES), i have created a Flips wrapper that will store the NES header inside the .bps manifest file. It's posted in the Discussion area if you wanna check it out.

Altho, I'm not sure I'm entirely "sold" on this particular strategy as there are many other, equally valid strategies to handle this issue: And this whole problem area arises because there are so many different source roms floating around out there in the wild, & we don't know if the User has iNes1, iNes2, Signed, Unheadered.

So, we could just as easily create patches against a completely "zero'd out" header: This ensures that every single bit is overwritten & correct (if the source rom was wrong).

Or, I'm thinking it might have just been better to simply ship with a Header file on patch creation: Since this strategy would be useful for other methods of patching (.ips, .ups, .rup, .xdelta, etc) rather than only working for .bps patches (eventho, they're arguably the best patching format ? bps or xdelta ? lol).

[i30817]

Haven't read the full comment yet, but romhacking.net will not have the full hack (and especially not if it's the hack creators uploading) ever because they don't want to be accused for "pirating" even if they never distribute the game, like happened to yuzu for instance. And I recommend any romhacking.net replacement not to go that route too, if they don't want to get hit by a 1 million lawsuit from a bully company and american oligarch courts.

[Europia79]

That's why i said they generate patches: We might be off-topic here, but there's no legal issue for distributing patches. ...I think the disconnect & misunderstanding here is that if you've done as much patching as me (literally thousands upon thousands), then you're going to encounter the scenario where the Author completely messed up on patch creation: And that can be an absolute nightmare, depending on System & Patch Format.

In fact, your whole post & "issue" arises because of the lack of any patching standards on RHDN (with regards to headers), which you even admit to them having incorrect info that was just copy & pasted from redump.

This is really an instance & an extension of the XYZ problem: Which, you (& others) have an "issue" that in your mind, you want solved in a particular way... But I'm saying that there are many other strategies & solutions.

Reminds me of a plugin conflict back in the day which lead to differing viewpoints as to which plugin had the responsibility to "solve" the conflict. Same deal here: A lot of people just assume the position that it'd be easy for all the Patchers to continually add on new features for various issues & use-case scenarios: But what I'm saying, is that in a server-client architecture of automated patch generation, this particular "issue" really does become a non-issue (legally as well).

[Europia79]

TLDR; Anyways, my whole point (since this issue is so old) is that an alternative solution would be to automate the remakes via scripting.

[virginviolet]

I don't think Flips is the best way to solve it, and I don't think header size options is the best solution available within Flips (it'd have quite poor user experience, you'd have to guess possible header sizes until one works).

A worse user experience is if you patch a rom and it doesn't work and you don't know why. If the resulting rom doesn't work, the user might have a look at all the options in the Flips UI and see if there is something they could do differently, and then try again.

My immediete UI solution would be to have a checkbox next to "Apply patch" by which you could select if the patch expects headered or unheadered rom, and it would be unchecked (no header) by default (since I think that is most common?). Or a dropdown. But maybe it could be hidden away somewhere. I'm sure there is a solution that doesn't infringe on user friendliness too much and doesn't look ugly.

I recently came across a 10 year old hack that just said "apply to the USA rom", no checksum or info about rom header. It didn't work with the NoIntro rom, and I was about to give up, because I didn't know what to do.

I'm glad I came across ipsbehead here thanks to @i30817!

When you patch a rom at runtime with bsnes-hd, it asks you whether or not the patch expects a headered or unheadered rom. That's nice. Even if that perhaps wouldn't be the best solution for Flips.

bsnes-hd screenshot


[Alcaro]

Okay, you have a point. I remain convinced that Flips is a bad place for this stuff, but you're right that every other location is also bad. (And so is the entire IPS format.)

I don't want a checkbox before choosing the patch - BPS users don't need to know about headers - but a message box after choosing the patch and ROM would be reasonable. (Only if IPS, and .sfc or .smc - other consoles don't have this problem, right?)

And I do like how bsnes-hd's prompt is worded.

(Doesn't solve the problem that I'm a lazy piece of shit, though.)

[i30817]

.pce too, IPSbehead also works for those and they're in a similar situation where the console doesn't use the headers, only the copiers and the size is the same and some romhacks make the mistake of including a header iirc.

Not NES roms, since those headers are necessary so removing that from patches can and usually does have side effects. Doesn't help that no-intro went lawful stupid there and listed the checksums for the NES roms without the headers but the actual files have them (to work), so romhacks usually either list the true checksum, which isn't the one in the .dat or the 'fake' one, which can cause patches to be applied to other files with IPS (and often break, naturally).

[i30817]

I don't know of other consoles with unneeded headers using ips for patches, but I wouldn't be surprised if there were some more. The main problem with this idea is the user using the option when they're wrong about the nature of the patch or the nature of the rom, so it would be great if there were sanity checks. But I'm a bit resigned to them being impossible knowing the ips format that doesnt even know the size of the file it applies to iirc. But maybe something can be done to check if the rom file is actually unheadered for those two\three rom formats.

[virginviolet]

I just want to add here that Rom Patcher JS allows you to add a temporary header, in case you have a no-header rom and the IPS expects a header rom. A checkbox (ha!) appears when you load in a headerless rom, a checkbox that allows you to add a temporary header. It gave me an output rom that was hash identical to ipsbehead+Flips output.

Rom Patcher JS screenshot


I don't want a checkbox before choosing the patch - BPS users don't need to know about headers - but a message box after choosing the patch and ROM would be reasonable.

Oops, I forgot about other path formats.

[Alcaro]

But it does not allow you to remove the header, if your ROM is headered and the patch is not?

Anyone mind if I accuse those headers of being a giant pain in the ass?

Either way, IPS headering and PCE headering sounds like two separate issues to me. If we teach Flips about PCE headers, that's gonna apply to BPSes too.

[i30817]

Honestly the only reason to add a header to a snes or pce rom these days is to apply or softpatch a ips patch... Usually a old one that is unfortunately using good tools dumps as base, even if it could not.

That's the reason I use IPSbehead instead of nsrt or similar, it's a lot less steps if you want to keep the original rom when softpatching.

Behead the ips and rename it as a softpatch (best case if the target rom has the same checksum as the no intro rom after removing any header), versus adding the header, applying the patch, removing the header, creating a new patch from the original to the copy and deleting the extras. When the base is a different rom (that is a different good tools dump than no intro), this slightly changes to 'track down the right dump, apply the patch, remove header, create a patch from the original no intro to the patched game, delete extras', but that's the worst case. This is for snes and pce ofc, for other consoles like nes remove all parts referring to adding and removing headers.

I do this dance to not have not no-intro roms when verifying the collection or adding to RetroArch.

[virginviolet]

But it does not allow you to remove the header, if your ROM is headered and the patch is not?

It does, actually.

Rom Patcher JS screenshot


Too bad I prefer Flips.

[i30817]

Either way, IPS headering and PCE headering sounds like two separate issues to me. If we teach Flips about PCE headers, that's gonna apply to BPSes too.

They're the same size as snes headers 512 bytes iirc. That's why IPSbehead works for both decaping ips meant for pce and sfc roms, it doesn't actually check anything about the patch target iirc the code (it's been a few years). I think it can't because ips is just that bad a format (you could because you'd actually have the intended patch rom in this scenario), and I suspect, not know, that it just moves the targets for modifications 512 bytes down or something like that.

I think that there were some cases where it could break a patch even if it was supposed to be for a headered rom, in case the modification was in the header (what happens, I Don't know, just that I reported it to a romhacker and they told me it was solved when they removed the accidental header modification the patch was doing).

It's actually a bit of unfortunate history that both pce and snes sets have a lot of hacks targeting headered roms with headers of the same size that are useless, so IPSbehead works to fix both. It comes from dumper hardware evolving from NES dumpers and doing the least effort possible I think. Most new hacks will use the no-intro set rom as target and will not have this problem even if they use ips.

[Europia79]

Bro, .smc & .sfc files are a complete non-issue, as you only need to apply the patch TWICE & see which one works.

The real "Header Issue" comes into play with .nes files (easily identifiable with the magic number 4E 45 53 1A), because these 16-byte Headers are actually NEEDED to run the hacks, and because there are near infinite amount of possible Headers for the Baserom: Now-a-days, what you'll most commonly run into are either (1) iNes v1.0, or, (2) iNes v2.0 Headers.

But that's not always the case: Like, for example, if the Author has been hacking & collecting for a very long time, then it's completely possible that they also started with an old rom that has an old, outdated Header, which they hacked & tested against, and which the Player also needs to run the game.

Thankfully, some Authors have noticed this issue & hence ship their romhacks with a 16-byte NES Header.

There are also what are called "SIGNED Headers", which are mostly iNes v1.0 which have been "signed" by the Dumper with their pseudonym.: Which, while NOT required to run the game, they are required if you're verifying hashes & checksums (to ensure that you started with the correct input... because you've verified the output hash).

IMO, probably the "best" solution is auto-generation of all these Hashes on patch creation (of both input & output roms), as well as saving the original 16-byte '.nes' Header that was used.