Versioning Extensions for the GRF File Format

[Garhoogin]

I propose here the idea of including file versioning information in the GRF file format to allow for file format extensions.

The idea I had in mind for such a system would be the addition of a new block type with the signature XVER. Decoders may identify the presence of this block to signify that the file is encoded using a newer encoder version, and to be able to either choose to handle or reject such files. My idea for this block is to have a 2-byte format version (high and low version), followed by an optional encoder ID string. The layout would be such:

Offset	Type	Name
0	u8	Major version
1	u8	Minor version
2	u8	Encoder ID length
3	char[...]	Encoder ID
		padding to multiple of 4 bytes...

Here the major version should be incremented for any breaking change to the file format that an older decoder should reject. Ideally, this block would come before the HDR block to signal to the decoder early on what format of data it should expect to see. I here propose that the file version to use with the BlocksDS version of the format should be at least 2.0.

The purpose of the Encoder ID would be to identify what software was used to produce the file. Since this is not strictly necessary for a runtime application to decode the file, I've made it optional in this outline. It would be a counted string of up to 255 octets in length, in no particular encoding.

An example for the data block using grit as an example encoder:

00000000:  58 56 45 52 18 00 00 00 02 00 12 67 72 69 74 20   XVER.......grit 
00000010:  76 31 2E 36 2E 30 2D 62 6C 6F 63 6B 73 00 00 00   v1.6.0-blocks...

I believe this would allow the format to be more easily extended, as no such system is currently in place with the format the way it is.

[AntonioND]

I think adding a version chunk makes the format more complicated than it has to be. I think it's enough to have a uint8_t that is incremented with each version that breaks compatibility. This value can be added to the "HDR " chunk, but the current "HDR " chunk isn't the same as the original one pre-BlocksDS, so I can just rename "HDR " to "HDR2".

I wouldn't add information about the tool that has generated the file, it's unnecessary information that makes the binary file grow for no good reason. On the DS we should really aim to be small in this case.

[Garhoogin]

My thought was that if you really wanted small files, you'd probably be using a better format anyways, and a compressor a little more intelligent than grit's. But I can understand where you come from. A simple u8 in the header would probably suffice.

[AntonioND]

A better format like what? GRF looks pretty compact to me, but I may be missing something.

I think I'm just going to switch from "HDR " to "HDRX" and add a version number to the chunk. I'll do this tomorrow, sorry for not being very responsive this week!

[asiekierka]

If you want to preserve backwards compatibility, repurpose the high byte of one of the Attr values:

    uint16_t gfxAttr;  ///< BPP of graphics (or GRFTextureTypes). 0 if not present.
    uint16_t mapAttr;  ///< BPP of map (16 or 8 for affine). 0 if not present.
    uint16_t mmapAttr; ///< BPP of metamap (16). 0 if not present.

If you don't, make the version number at least an uint16_t. However, I am somewhat unconvinced by the idea, as there's no central authority on which version numbers are occupied by which encoders or version formats - any fork of Grit, or even dkP's upstream, could easily disagree with us on that regard. What if two different version 3s appear?

[AntonioND]

@asiekierka That's why I want to use a completely different header chunk ID. If you don't find the ID you're expecting, you're using the wrong version.

[AntonioND]

https://github.com/blocksds/grit/pull/7

https://github.com/blocksds/libnds/pull/131

Draft.

[Garhoogin]

A better format like what? GRF looks pretty compact to me, but I may be missing something.

I think I'm just going to switch from "HDR " to "HDRX" and add a version number to the chunk. I'll do this tomorrow, sorry for not being very responsive this week!

I suppose "better" probably isn't the right word. I meant to say that there's a fair bit of overhead I the format that if someone were worried about file size, they shouldn't be using this. But I do imagine that the whole version block could be a bit much.

@asiekierka That's why I want to use a completely different header chunk ID. If you don't find the ID you're expecting, you're using the wrong version.

I feel different block IDs would do a good job of not confusing someone else's decoder. It's probably not super important that the format be strictly binary compatible, since people can just re-convert their assets when upgrading their tooling.