Increase support for HMD models

[trigger-segfault]

Short explanation

Support for HMD has been majorly revamped. A ton of bugs were fixed that either resulted in incorrect positions, or invalid data and exceptions.

All LEGO Rock Raiders (PSX) HMD models (that I've been given) are now supported (thus fixing issue #47). Some models seem to have issues with tiling (refinery.hmd) but these may be issues with the models themselves vs. with the reader, as they are unused models. Some models (rockmons.hmd and rockwhal.hmd) have image data that's pointed to past the bounds of the file data, but the model data itself is still valid.

Preview: A rock monster HMD model with limb attachments


Changes and additions

• Types are now read at the correct position by seeking after reading type data. dataCount and dataSize were fixed to be read in the right order.
• Coord and Matrix reading has been fixed to read correct padding, use correct divisors, and 1-index comparisons with TMDID.
• Shared Geometry data support has been added. This is a form of attached limbs that doesn't use attachments to real triangle vertices.
• Image data is correctly loaded, and loading has been fixed to handle the correct pmode. Out of bounds errors reading image data will not abort reading the HMD file, since valid model data may still exist.
• All HMD polygon packets are now supported (with exception of presets and strip meshes).

What hasn't been fixed

• Culling: Some models are flat polygons with different textures on each side. Culling (Front only) needs to be enabled in the render options to view these properly. For now I have not implemented this because it's unclear how many other things I could break by assuming culling should be enabled by default. I know TMD and HMD models have BOT flags for single or double-sided polygons, but again. Adding support for specific model render settings is too messy for this already-large commit.
• Semi-transparency: A handful of models in LRR (like glowing rectangles, jets, propellers, and lasers) use a Semi-transparency mode that's also a mesh-specific render option. I've also experimented with getting this working (which requires a separate pass for these meshes), but it's the same issue of it requiring too many invasive changes to include in this commit.
  • By extension, not fixing semi-transparency will keep the issue where certain black surfaces (with the stp bit set in the clut pixel) will be hidden when they shouldn't be.

---

Detailed explanation

HMDParser

ParseHMD

• Coord needs to be 1-indexed when compared to the model TMDID (modelEntity.TMDID == c + 1). This is documented on page 98 where "Pointer to primitive 1" is next to "Pointer to primitive of coordinate 0" and so on. The first primitive is a pre-process.

ReadCoord

• Switch matrix muliplication order: superMatrix * worldMatrix → worldMatrix * superMatrix. Many models were giving inaccurate results with the original order, but all look correct after switching.

ReadMatrix

• Change divisor for translation int32s from 4096f to 65536f (0x10000). Again, many models with lots of parts were giving horribly innacurate positions without this change. I couldn't find documentation stating this is what the divisor is, but it makes logical sense for half of the bits to be dedicated to the fraction.
• Add 2 bytes of padding after the translation int32s. I found this out from a few GitHub sources documenting PSX structures (all with the same exact info). Matrixes are 32 bytes long, but are only 30 bytes without the padding, thus the second matrix, super index, and all future matrices will be read incorrectly.
  • Although the struct example and common sense says that the padding should appear after the odd number of int16s, and before the int32s, testing showed that the padding needed to come after the int32s, otherwise most models ended up centered very close to (0, 0, 0).

// size: 0x20
struct MATRIX {
    // offset: 0000 (18 bytes)
    short m[3][3];
    // offset: 0014 (12 bytes)
    long t[3];
};

ProccessPrimitive

• Swapped names for dataSize and dataCount (because they were read in the wrong order). The size is in the lower 16 bits, which comes first with Little Endian data.
  • Both ProcessNonSharedGeometryData and ProcessMimeVertexData correctly passed the previously-named "dataSize" (dataCount) when they were asking for dataCount.
  • But ProcessGroundData passed the previously-named "dataCount" (dataSize) / 4 when asking for dataCount, so my assumption is that original tests were forced into finding a way to transform dataSize into the expected value. I have not changed this parameter to use the newly-named dataCount because I have zero HMD files with Ground Data to test against. However, I have left a comment explaining the issue, for future programmers that are debugging issues with the ground data.
• Added support for shared geometry data category (another way of handling limbs). A more helpful explanation of what Shared Geometry does was found documented on page 294. It basically behaves similar to the existing Attached Limbs functionality in PSXPrev, except the attachments are not bound to a specific vertex of a real triangle. Instead, attachments are bound to the coordinate of the current block (primitive index). So to handle this, model entities (and triangles) now hold extra data for storing shared geometry.
• Image Data now properly checks if category == 2. Originally it was if category == 1 right after the same comparison was done for shared geometry data.
• Types now seek to the end of their data at the end of each loop cycle. This fixes issues where unsupported categories (and supported categories that seek back to the start, i.e. Image Data) would leave the reader at a position too far before the next type started, causing invalid data to be read. Note that the offset we start from is before dataSize/dataCount are read, and also that dataSize is defined in units of 4 bytes. Also note that dataSize defines the whole size, and not the size of each element in dataCount.
• Handling for shared vertices and normals has been added to the end of the function. Either the shared geometry is stored in an existing model (if a non-shared model was read in this block), or adds it to a dummy model with 0 triangles which only contains attachable information.

ProcessImageData

• Added ProcessImageDataPrimitiveHeader at the beginning of the function to grab the real imageTop and clutTop positions.
• Both old imageTop and clutTop variables were renamed to imageIndex and clutIndex. These are still muiltiples of 4, but they also need to be added to their respective top variables to get the full position.
• Changed the passed pmode argument for ReadPalette and ReadTexture. It turns out that both 16cluts and 256cluts are supported, and the only way to distinguish these is by combining the clut dimensions. pmode is set to 0 if clutWidth * clutHeight <= 16, otherwise pmode is set to 1 (asside from when !hasClut, in which case pmode is set to 3).

ProcessNonSharedGeometryData

• Added an argument bool shared to determine which category called this function. This differentiates which header to read, how to process vertices and normals, and tells AddTrianglesToGroup to setup attached indices. This is necessary because shared geometry data will read standard geometry packets on the last block of the HMD file.
• Switched to using added-functions ReadVertex and ReadNormal, since there's now multiple places where these are needed.

ProcessSharedGeometryData

• Function added to process shared geometry pre-calculation data.
• Shared vertices and normals are read into dictionaries which are used to lookup attachments by index.

  ---

TMDHelper

CreateHMDPacketStructure

• Documented each flag, and also changed code to be read like an integer and then assigned to a quad boolean if equal to 2. This makes it clearer that code is not just 2 different values.

ParsePrimitiveData

• Added the following arguments:
  • bool hmd - This is needed because some behavior when reading packets differs between TMD and HMD. HMD generally has more compact structures and avoids excess padding, it also reads colors before UVs.
  • bool tiled - This is needed for HMD packets with tiled information. I'm not entirely sure if textures are being tiled correctly in the VRAM or during the rendering process, but reading this information is necessary regardless.
• Tiled information is now read before all else.
• Order of reading colors before or after UVs has been changed depending on whether the reader is an HMD.
• Behavior of end padding after vertices has been changed for HMD, since this differs from TMDs in many cases.
• HMD behavior for reading Normal0, Vertex0, and/or Vertex1 early (during the UVs 2 and 3 padding) has been added. This also handles excluding reading of these data points during the vertex/normal section if already read.
• With these changes, all known packet types for HMDs (besides presets and strip meshes) should now be supported.

AddTrianglesToGroup

• Added argument bool attached to handle assigning attached indices for vertices and normals. This is needed for HMD shared geometry, since there's no callback function to know when a triangle is added, and additionally this function needs to know if normals were applied.
• Fixed debug logging of the primitiveData dictionary so that the full contents of the dictionary isn't printed Count times, just one time.
• Added parsing for new TILE PrimitiveDataType, which contains four parameters: tum, tvm, tua, and tva. These parameters are then used in TileS and TileT functions around all S/T values. Note that the S/T values will stay the same if there is no tiled information.
• AttachedIndices and AttachedNormalIndices are now assigned to clones of the original indices arrays if attached is true (and if hasNormals is true for normals).

---

ModelEntity

• Added dictionaries for storing shared geometry information that isn't tied directly to existing triangles. This is needed to work in shared geometry behavior, and this was the cleanest method I could come up with.
• Added handling for shared geometry linking to FixConnections.

Triangle

• Added array for storing shared geometry normal indices. This is used in ModelEntity.FixConnections.

TIMParser

• Added argument bool allowOutOfBounds to ReadPalette and ReadTexture to ignore out-of-bounds exceptions when reading a file. This is only set to true when reading for HMD files, since some files have been encountered with invalid image data, but valid model data.
• ReadPalette has been refactored to handle possible scenarios where the dimensions are less than the total size of the palette.

[rickomax]

Man, you are awesome! Thank you for the efforts! Merging it now!