Support for non-PVRT chunks in PVMs

PiKeyAr commented 3 years ago

Hello,

I was trying to open a PVM extracted from one of the archives in Sonic Shuffle and ran into an error. There are two textures in the file, but the first texture isn't loaded correctly because the PVM contains additional metadata before texture data begins. Here's the PVM: shf.zip In Archive Explorer the first texture is only 24 bytes, but that is the length of the metadata chunk, and the actual texture is right after it. The metadata begins with the MDLN header. It appears to have been used by Sega's PVM converter tool included in Katana SDK. Apparently there were also other headers:

/*
 * texture chunk name
 */
/* pvr */
#define iff_PVRT    'PVRT'     /* PowerVR texture             */ 
#define iff_GBIX    'GBIX'     /* globalIndex                 */ 
#define iff_PVRI    'PVRI'     /* texture Info                */ 

/* pvm */
#define iff_PVMH    'PVMH'     /* PVM hedder                  */ 
#define iff_COMM    'COMM'     /* Comment                     */ 
#define iff_CONV    'CONV'     /* PVM Converter               */ 
#define iff_MDLN    'MDLN'     /* model name                  */ 
#define iff_IMGC    'IMGC'     /* Image Container             */

Would it be possible to make the archive reader skip this data so that it extracts the textures correctly?

nickworonekin commented 3 years ago

Looks like we can reliably detect if this chunk is present by checking if 0x10 is set in the flags byte at 0x8. Added the enhancement label as we should allow people to add model information.

PiKeyAr commented 3 years ago

Yeah, I also came to that conclusion about the flag. I'm not sure how to detect other metadata chunks though, and I haven't found any PVMs yet that have them. I also noticed that Sega's pvmconv tool sets the byte at 0x09 to 01, which probably means something too. I think our best bet is to try those ancient converters from the SDKs for old 3D model tools to see if they can export this metadata, and then try to build a PVM with it. I think it would be cool to be able to view and extract this data, though it doesn't seem like there's any practical use writing it

nickworonekin commented 3 years ago

I played around with pvmconv and was able to eventually get each of those chunks listed there present in the PVM. This is what I observed:

MDLN chunk

Contains one or more model filenames (whose textures are associated with this PVM?). 0x10 is set at byte 0x8 when this chunk is present.

Chunk structure:	Offset	Length	Description
0x0	4	'MDLN'
0x4	4	Length of chunk minus 8

Repeat for each entry within the chunk:	Offset	Length	Description
0x0	Varies	Filename (including null terminator)

PVMI chunk

Contains the original filenames of the textures before they were converted and added to the PVM. Doesn't appear to be present if a PVR was directly added to the PVM. 0x20 is set at byte 0x8 when this chunk is present.

Chunk structure:	Offset	Length
0x0	4	'PVMI'
0x4	4	Length of chunk minus 8
0x8	2	Total length of each filename (including null bytes). In general, the length of the longest filename.
0xA	2	Always 2

Repeat for each entry within the chunk:	Offset	Length	Description
0x0	n	Filename
n	2	Always 32

CONV chunk

Appears to be the name of the converter used to convert the textures. For pvmconv, it's the name of the executable sans extension (so just pvmconv). Doesn't appear to be present if a PVR was directly added to the PVM. 0x40 is set at byte 0x8 when this chunk is present.

Chunk structure:	Offset	Length
0x0	4	'CONV'
0x4	4	Length of chunk minus 8
0x8	Varies	Name of converter

IMGC chunk

Contains the original texture data of the converted textures. This chunk is repeated per converted texture. Doesn't appear to be present if a PVR was directly added to the PVM. 0x80 is set at byte 0x8 when this chunk is present.

Chunk structure (repeat for each converted texture):	Offset	Length
0x0	4	'IMGC'
0x4	4	Length of chunk minus 8
0x8	2	Always 0
0xA	Varies	Original texture data

PiKeyAr commented 3 years ago

Very nice! I've been looking at the Japanese manual for pvmconv, and it appears that it could also add the following chunks: PVPL - PVP file but embedded into the archive PVPN - PVP file name I've also found some info on the PVRI chunk but it's really vague and incomplete. So far it confirms my assumption that unlike other chunks it was meant for each texture rather than globally for the whole PVM.

PiKeyAr commented 3 years ago

Here are all PVM flag bits (referred to as pvmstatus in the SDK). Apparently it's part of the Ninja2 specification, so some of it may not be relevant to older PVMs.

The PVMH_PS_ENTRYINFO flag indicates the presence of the so called EntryStatus part for PVM entries. In addition to texture dimensions it can store a variety of flags:

The file N2pvmspec.pdf in DC SDK 2.00J contains detailed descriptions of each PVM chunk in Japanese. The structure for the COMM chunk is as follows:

Offset	Length	Description
0x0	4	'COMM'
0x4	4	Length of the comment string
0x8	Varies	Comment string

The COMM chunk can be placed anywhere in the PVM.

nickworonekin commented 3 years ago

The issue with reading PVM archives that contain these chunks should be fixed now; the other chunks will simply be skipped over when parsing the PVM. Some more work could be done to better support these (for example, being able to retrieve data from these chunks or support adding them), but as of now archives containing them will no longer cause issues.

PiKeyAr commented 3 years ago

Great news!

It doesn't seem like these chunks are read by the games, so not being able to read or write them isn't a big deal. The PVM from Shuffle works fine now. Thanks a lot!

nickworonekin / puyotools