Add a 'write to stdout' mode

[i30817]

(and maybe a 'read from stdin' mode, though that one is not so important).

This is in order to not be forced to create tmp files. In my usecase i'm using flips to help create .dat files for RA.

In these (variably huge) files i'd like to apply the patch to the rom and grab stdout in order to calculate the needed checksums without actually writing the file to disk at any time.

xdelta has something like that iirc:

https://github.com/jmacd/xdelta/blob/wiki/CommandLineSyntax.md 'use stdout'

Now that i think about stdin is (when supported) in linux indicated by a '-'. It's probably not very possible here because patchers need two input files (though it's totally possible in bash with process substitution, which is annoying).

[Alcaro]

If you want the crc32 from a BPS or UPS patch, read the last 12 bytes, they contain source crc32 then target crc32 (then the patch's own crc32, for whatever reason). Far faster than patching, and doesn't even need the source ROM present.

If you want IPS or another checksum, use /dev/stdout as output file. It currently fails on the GTK frontend (CLI works), but that's a bug. I'll fix it later today.

Not sure about Windows, and I no longer have the ability to test anything there. Do you need that, or is /dev/stdout enough?

[i30817]

1. I need more than one version, md5, sha1 etc. This is necessary to create a valid RA dat entry.
2. I want to verify the patch actually applies, because i've seen plenty of bad ones (ips mostly).
3. some of the patches are ips.

But that trick is neat, i'll probably use it when possible (verify source on dat when patch is bps, probably).

/dev/stdout... not sure. Even if it works, i'd prefer it worked on windows too... I'll try it but even if it does work, i'd prefer a solution that's portable upstream.

[i30817]

btw i've read around the net that bps skips headers in some roms, does this apply to SNES only? Or to NES too?

Including the footer crc32?

[Alcaro]

Yes, if you're using IPS or want things other than CRC32, then the last 12 bytes won't help you.

And yes, stdout does contain a success message.

But using /dev/stderr as output should work (once I've fixed the GTK front).

flips --apply foo.ips bar.sfc /dev/stderr 2>&1 >/dev/null | md5sum

[Alcaro]

The header skipper is SNES only, and applies to the 512-byte SMC headers on the input files, not the headers of the BPS patch itself. As far as any other patcher knows, it's a perfectly normal BPS patch that turns one unheadered ROM into another.

You can turn it off with --exact, but that's kinda pointless when applying. It's autodetected whether the patch is intended for a headered ROM.

[i30817]

Can i use a FIFO? I mean if i provide a FIFO (named or not) that 'exists' already as a 'file' to the cmd-line output parameter of flips, does it write to it 'as is' or tries to remove it first and make a new file?

stderr works, but if i can use something more portable (i'm using python subprocess) i don't see why not.

[i30817]

edit: never mind problem between keyboard and chair. Source already had a header.

[Alcaro]

I expected GFile to overwrite that, but testing reveals that it works. Unexpected, but sure, why not. (If it hadn't, I'd tell it to try writing with libc before GFile.)

CLI and Windows fronts will write to the intended target as well. Named pipe is probably your only option on Windows, there's no /dev/stdout there. (No, CONOUT$ doesn't count.)

If you apply an unheadered BPS to a headered SNES ROM, Flips will assume you want headered output, and will copy it unchanged from the source. In this case, the easiest fix would be applying to unheadered ROMs.

If you've got an IPS that turns a headered source into a headered target, you have to give Flips a headered input, or you'll get a corrupt file. IPS patches don't contain any information about which file it's for, so Flips just assumes the input is correct.

If you've got an IPS that turns an unheadered source into a headered target (I've seen a few of those), Flips is emitting the correct target file, and I'm not changing that. You'll have to remove the header after patching.

If the inputs are FIFOs too, Flips won't like that, size+read is way easier than looping a read and reallocating until EOF. Especially when I have to do it thrice (GTK, Windows, libc). I'd accept a PR, but I'm not doing it myself.

If you're trying to ask anything else, please clarify. Is Flips writing the SMC header after the actual ROM contents?

[i30817]

No no, it was me being stupid. BPS was not guilty, it was just me forgetting i had added a header with nsrt to see how it would behave. Turns out it behaves logically, keep a header if one existed on the source, don't add one if it doesn't.

About ips and headers, i take care that whenever i see one that needs a header on romhacking, i apply ipsbehead at them. Such a simple program, but so useful. It's a pity that this operator requires metadata otherwise i'd request the feature in flips too, so i could stop caring about doing it myself.

Speaking of that i think i finished the program...

https://gist.github.com/i30817/c5332248f46113fcb4ca03081f7673f2

Not to convert all my 'revert' xdelta patches to bps so i can propose dat files to psp games for redump...

[Alcaro]

Neat tools. Yeah, the extreme simplicity of IPS is useful sometimes. That specific operation seems a little too rare (and too different from existing operations) to include in Flips, but keeping it as a separate tool is a good idea.

(except for SNES headers, which is the one header bps ignores when creating/applying a patch)

Floating IPS drops SNES headers. The BPS spec itself does not demand such a thing, and other BPS creators leave headers alone. I'd prefer if I could remove that feature, it is a pretty ugly hack, but I had to make that compromise for the SMW hacking scene to switch from BPS.

Either way, if that tool exists, I'll just assume you're happy with how stderr/fifos work.

[i30817]

Yeah i'm happy now that i'm using FIFOs. I made a page and pypi install for it: https://github.com/i30817/rhdndat

BTW, there is another '512 bytes' header situation on another hacking scene, but it's mostly with IPS, don't know how BPS would react. It's turbografx 16 / Pc-Engine patches. It's so similar, that i can use ipsbehead on headered ips patches just like in SNES headered ips patches.