search

kitlith / sims3-rs

Command-Line Tools and Libraries for sims3 package files
MIT License
2 stars 0 forks source link

Just Citing/Acknowledging Your Work #3

Closed anonhostpi closed 2 months ago

anonhostpi commented 2 months ago

Gonna open then immediately close this issue. Just reaching out to let you know that my DBPF wiki cites this repository.

I'm avidly working on my own DBPF reader. If you see anything missing from my wiki (...or my lib), feel free let me know! I'm avidly working on them every day

I just wanted to be sure you knew I was citing your work! Very nice tool by the way! It's nice to see a mod tool for Maxis games written in Rust!

kitlith commented 2 months ago

Neat! It's a shame that I haven't touched this in a while.

anonhostpi commented 2 months ago

I like the idea of one written in Rust. Would be insane if OpenTS2 was ported to Rust. It would probably be the fastest sim on the planet.

kitlith commented 2 months ago

re: "anything missing": I have this memory of running into a chunk that neither my tool or S3PE were successfully parsing, until I took a closer look and made some guesses. However, I don't appear to have made any comments about it, so I can't point out what, exactly, it was. I know I was planning on updating the wiki around that time, too, but I don't remember if that actually happened.

It was probably something I ran into while writing geom_tri_tool, though.

anonhostpi commented 2 months ago

Was it a save file? s4pe had an interesting way of handling DBPF v2.0 entries. Basically, they discovered a header entry that was a partial one. The few fields this entry had were shared across all entries.

For save files, I believe this was the group ID.

I think the first 4 bytes after the index offset (v2.0 offset, not v1.0 offset) tell you which fields the header entry contains.

You have to translate these bytes (little endian) to binary, then do basic binary flag math.

0b0011 means the type ID (first 4 bytes) and the group ID (second 4 bytes) are pulled from the header entry.

According to s4pe, only the first 4 entry fields are shared/shareable, but technically, according to the math, all DBPF v2.0 fields can be derived from this header entry.

kitlith commented 2 months ago

Basically, they discovered a header entry that was a partial one. The few fields this entry had were shared across all entries.

I think that was old news by the time I was working on this, and I wrote all my code regarding the header entries according to the wiki.

Looking in my source directory, I think it was a GEOM entry inside an RCOL, given that I appear to have a couple geom chunks extracted as individual files, named some_geom and weird.geom, as well as an imhex project inspecting weird.geom with a pattern thrown together to figure out what was going on:

#include <std/core.pat>
#include <std/ptr.pat>

struct VertexAttribute {
    u32 data_type;
    u32 sub_type;
    u8 bytes;
} [[static]];

fn vertex_size(ref auto attributes) {
    u32 sum = 0;

    for (u8 iii = 0, iii < std::core::member_count(attributes), iii += 1) {
        sum += attributes[iii].bytes;
    }

    return sum;
};

struct SubMesh {
    u8 index_sz;

    u32 index_count;
    u8 index_buffer[index_count*index_sz];
};

union Padded<T, auto sz> {
       padding[sz];
       T inner [[inline]];
};

struct MTNFEntry {
    u32 param_id;
    u32 data_type;
    u32 data_size;
    u32 data_offset;
};

struct MTNF {
    u8 magic[4];
    padding[4];
    u32 data_size;

    u32 count;
    MTNFEntry entries[count];
    u8 data[data_size];
};

struct TGIEntry {
    u32 type_id;
    u32 group_id;
    u64 instance;
};

struct TGITable {
    u32 count;
    TGIEntry entries[count];
};

struct TGIRef {
    u32 tgi_off;
    u32 tgi_sz;
    Padded<TGITable, tgi_sz> tgi @ tgi_off + $ - 4;
};

struct GEOM {
    char magic[4];
    u32 version;

    TGIRef tgi;

    u32 embedded_id;
    if (embedded_id != 0) {
        u32 mtnf_sz;
        Padded<MTNF, mtnf_sz> mtnf;
    }
    u32 merge_group;
    u32 sort_order;

    u32 vert_count;
    u32 attrib_count;
    VertexAttribute attributes[attrib_count];
    u8 vertex_buffer[vertex_size(attributes) * vert_count];

    u32 item_count; // ???
    SubMesh items[item_count];

    u32 skin_index;
    u32 count1;
    u32 bone_hashes[count1];
};

GEOM file @ 44;

I'll compare against the wiki and see if I can pinpoint what the issue was.

anonhostpi commented 2 months ago

I just started my dig into GEOMs, so I don't have a lot of info yet, but I can get to work here in a few hours.

kitlith commented 2 months ago

I think I found it. From the wiki: http://simswiki.info/wiki.php?title=Sims_3:0x015A1849

DWORD ItemCount                 // will usually be 1 (hardcoded in pipeline exporter), but sometimes other numbers
--repetition ItemCount:
    BYTE BytesPerFacePoint

DWORD NumFacePoints
--repetition NumFacePoints:
    BYTE[BytesPerFacePoint] // Given that ItemCount is 1, BytesPerFacePoint is 2, this is a list of WORDs
                // Each set of three forms a face.

according to my pattern file and binrw parser, that last section should be indented, part of the previous repetition group.

like so:

DWORD ItemCount                 // will usually be 1 (hardcoded in pipeline exporter), but sometimes other numbers
--repetition ItemCount:
    BYTE BytesPerFacePoint

        DWORD NumFacePoints
        --repetition NumFacePoints:
            BYTE[BytesPerFacePoint] // Given that ItemCount is 1, BytesPerFacePoint is 2, this is a list of WORDs
                        // Each set of three forms a face.

naming on the wiki is a little weird, I'm pretty sure this is just a list of index buffers used for 3d rendering.

anonhostpi commented 2 months ago

The sims wiki doesn't seem to be maintained well, so I'm not surprised.

kitlith commented 2 months ago

It's moot if the 'item count' is 1, which it usually is. I only know about this because I wrote tools designed to inspect entire custom content collections, and someone stumbled across and sent me an example where the count wasn't one.

anonhostpi commented 2 months ago

I'm guessing that's for an object that has a subdivided mesh

anonhostpi commented 2 months ago

Do you remember anything else about the file? I'm assuming it was for TS3, but is it possible it was for a different game like Spore?

kitlith commented 2 months ago

It was for TS3. I still have the original package file, probably, though I'd need to check which one it was. I'd need to check what the sharing guidelines for that file are, though.

anonhostpi commented 2 months ago

Took a nap and slept for a bit too long. I think I'm going to abstract these with Kaitai, since it uses YAML

anonhostpi commented 2 months ago

Out of curiosity, do you know what the mergeGroup entry is supposed to do? I see a lot of docs on its existence, but not a lot stating what it does.

kitlith commented 2 months ago

no clue, off the top of my head

anonhostpi commented 2 months ago

Ok, I've got the geom entry defined in Kaitai right up to the point just before itemCount gets defined:

meta:
  id: geom
  file-extension: geom
  endian: le
seq:
  - id: header
    type: header
  - id: shader
    type: shader
  - id: merge_group
    type: u4
  - id: sort_order
    type: u4
  - id: vertex_collection
    type: vertex_collection
  # - id: itemCount # would go here
types:
  header:
    seq:
      - id: magic
        contents: 'GEOM'
      - id: version 
        type: u4
        valid:
          any-of: [0x00000005, 0x0000000C]
        doc: "see [Sims4Tools \\> GEOM.cs#L129](https://github.com/s4ptacle/Sims4Tools/blob/fff19365a12711879bad26481a393a6fbc62c465/s4pi%20Wrappers/MeshChunks/GEOM.cs#L129C67-L129C77)"
      - id: tgi_offset
        type: u4
        doc: Offset to the reference data from this object's offset -- may also be from this sequence entry's offset
      - id: tgi_size
        type: u4
  shader:
    seq:
      - id: id
        type: u4
        doc: "also referred to as 'EmbeddedID' in some tools. see [simswiki \> Sims 3 \> GEOM](https://simswiki.info/wiki.php?title=Sims_3:0x015A1849)"
      - id: size
        type: u4
        if: id != 0
      - id: mtnf            # Requires definition
        type: mtnf 
        if: id != 0
  vertex_collection:
    seq:
      - id: count
        type: vertex_collection_count
      - id: formats 
        type: vertex_format
        repeat: expr
        repeat-expr: count.formats
      - id: verteces
        type: vertex
        repeat: expr
        repeat-expr: count.verteces
  vertex_collection_count:
    seq:
      - id: verteces
        type: u4
      - id: formats 
        type: u4
  vertex_format:
    seq:
      - id: type
        type: u4
        valid:
          any-of: [1,2,3,4,5,6,7,10]
        doc: |
          1. Position
          2. Normal
          3. UV
          4. Bone Assignment
          5. Weights
          6. Tangent Normal
          7. TagVal
          10: VertexID
      - id: subtype 
        type: u4
        valid:
          min: 1
          max: 4
      - id: size
        type: u1
  vertex:
    seq:
      - id: elements
        type: vertex_element(_parent.formats[_index].type)
        repeat: expr
        repeat-expr: _parent.count.formats
  vertex_element:
    params:
      - id: index
        type: u4
    seq:
      - id: content
        type:
          switch-on: index
          cases:
            1: vertex_element_position
            2: vertex_element_normal
            3: vertex_element_uv
            4: vertex_element_bone_assignment
            5: vertex_element_weights
            6: vertex_element_tangent_normal
            7: vertex_element_tag_val
            10: vertex_element_vertex_id
  vertex_element_position:
    seq:
      - id: x
        type: f4
      - id: y
        type: f4
      - id: z
        type: f4
  vertex_element_normal:
    seq:
      - id: x
        type: f4
      - id: y
        type: f4
      - id: z
        type: f4
  vertex_element_uv:
    seq:
      - id: u
        type: f4
      - id: v
        type: f4
  vertex_element_bone_assignment:
    seq:
      - id: bone
        type: u32
  vertex_element_weights:
    seq:
      - id: weight
        type: f4
        repeat: expr
        repeat-expr: 4
  vertex_element_tangent_normal:
    seq:
      - id: x
        type: f4
      - id: y
        type: f4
      - id: z
        type: f4
  vertex_element_tag_val:
    seq:
      - id: value
        type: u32
  vertex_element_vertex_id:
    seq:
      - id: id
        type: u32
anonhostpi commented 2 months ago

Ok, this should be the full Kaitai Struct (per your spec):

anonhostpi commented 2 months ago

It's moot if the 'item count' is 1, which it usually is. I only know about this because I wrote tools designed to inspect entire custom content collections, and someone stumbled across and sent me an example where the count wasn't one.

This is how it is parsed in s4pi. Interestingly, they appear to have used NumFacePoints for itemCount:

anonhostpi commented 2 months ago

Oh, I think my Kaitai might be wrong. I'll need a copy of the package file you found, but I think I might know how indexCount and that array of bytes is supposed to be parsed.

kitlith commented 2 months ago

This is how it is parsed in s4pi. Interestingly, they appear to have used NumFacePoints for itemCount:

* https://github.com/s4ptacle/Sims4Tools/blob/fff19365a12711879bad26481a393a6fbc62c465/s4pi%20Wrappers/MeshChunks/GEOM.cs#L154-L171

In other words, they do the same thing as s3pe, and don't handle more than 1.

anonhostpi commented 2 months ago

Well they partially do, they just block it, if validation is enabled. Did you verify your pattern against that file? Did parsing it produce something sensible? I was thinking it could also be another form of a data formatter, not just an array length

kitlith commented 2 months ago

@anonhostpi

Well they partially do, they just block it, if validation is enabled.

They don't do anything with that item count aside from checking that it's equal to 1 if validation is enabled, which means that when validation is disabled it will silently read incorrect data, which is even worse than I previously thought. i'm 99% sure s3pe doesn't let you turn off erroring when the count is not 1, but it's also been months since I checked.

My file is a version 5 GEOM, btw, if that ever becomes relevant.

Modified GEOM ksy ```ksy meta: id: geom file-extension: geom endian: le # imports: # - path/to/mtnf.ksy seq: - id: header type: header - id: shader type: shader - id: merge_group type: u4 - id: sort_order type: u4 - id: vertex_collection type: vertex_collection - id: body_count type: u4 - id: bodies type: body repeat: expr repeat-expr: body_count - id: skin_controller_index type: u4 - id: bone_count type: u4 - id: bones type: u4 repeat: expr repeat-expr: bone_count doc: 32-bit hash of the bone name # insert TGI block list here types: header: seq: - id: magic contents: 'GEOM' - id: version type: u4 valid: any-of: [0x00000005, 0x0000000C] doc: "see [Sims4Tools \\> GEOM.cs#L129](https://github.com/s4ptacle/Sims4Tools/blob/fff19365a12711879bad26481a393a6fbc62c465/s4pi%20Wrappers/MeshChunks/GEOM.cs#L129C67-L129C77)" - id: tgi_offset type: u4 doc: Offset to the reference data from this object's offset -- may also be from this sequence entry's offset - id: tgi_size type: u4 shader: seq: - id: id type: u4 doc: "also referred to as 'EmbeddedID' in some tools. see [simswiki \> Sims 3 \> GEOM](https://simswiki.info/wiki.php?title=Sims_3:0x015A1849)" - id: size type: u4 if: id != 0 - id: mtnf # Requires definition type: mtnf if: id != 0 vertex_collection_count: seq: - id: verteces type: u4 - id: formats type: u4 vertex_collection: seq: - id: count type: vertex_collection_count - id: formats type: vertex_format repeat: expr repeat-expr: count.formats - id: verteces type: vertex repeat: expr repeat-expr: count.verteces vertex_format: seq: - id: type type: u4 valid: any-of: [1,2,3,4,5,6,7,10] doc: | 1. Position 2. Normal 3. UV 4. Bone Assignment 5. Weights 6. Tangent Normal 7. TagVal 10: VertexID - id: subtype type: u4 valid: min: 1 max: 4 - id: size type: u1 vertex: seq: - id: elements type: vertex_element(_parent.formats[_index].type) repeat: expr repeat-expr: _parent.count.formats vertex_element: params: - id: index type: u4 seq: - id: content type: switch-on: index cases: 1: vertex_element_position 2: vertex_element_normal 3: vertex_element_uv 4: vertex_element_bone_assignment 5: vertex_element_weights 6: vertex_element_tangent_normal 7: vertex_element_tag_val 10: vertex_element_vertex_id vertex_element_position: seq: - id: x type: f4 - id: y type: f4 - id: z type: f4 vertex_element_normal: seq: - id: x type: f4 - id: y type: f4 - id: z type: f4 vertex_element_uv: seq: - id: u type: f4 - id: v type: f4 vertex_element_bone_assignment: seq: - id: bone type: u4 vertex_element_weights: seq: - id: weight type: f4 repeat: expr repeat-expr: 4 vertex_element_tangent_normal: seq: - id: x type: f4 - id: y type: f4 - id: z type: f4 vertex_element_tag_val: seq: - id: value type: u4 vertex_element_vertex_id: seq: - id: id type: u4 body: seq: - id: face_size type: u1 - id: face_count type: u4 - id: faces size: face_size repeat: expr repeat-expr: face_count mtnf: seq: - id: magic contents: 'MTNF' - id: whatever type: u4 - id: data_size type: u4 - id: count type: u4 - id: entries type: mtnf_entry repeat: expr repeat-expr: count - id: data type: u1 repeat: expr repeat-expr: data_size mtnf_entry: seq: - id: param_id type: u4 - id: data_type type: u4 - id: data_size type: u4 - id: data_offset type: u4 ```
anonhostpi commented 2 months ago

Ah, thanks. Good eye. Always mix up bits and bytes.

anonhostpi commented 2 months ago

What's the markdown for hidden/spoilers again? I forget how to do that

anonhostpi commented 2 months ago

My file is a version 5 GROM, btw, if that ever becomes relevant.

I do have to make another modification to support version 0x0...0c. You may find this interesting:

It takes the place of skin_controller_index. Might be a new way to reference a skin controller

anonhostpi commented 2 months ago

Oh here we go, took some digging, but I found the updated format on the sims wiki:

https://simswiki.info/index.php?title=Sims_4:0x015A1849&oldid=68322

anonhostpi commented 2 months ago

Oh shit, I just realized the develop branch on S4PE is way further out than the main branch. There's 2 other GEOM versions discovered:

https://github.com/anonhostpi/Sims4Tools/blob/b5db166dd4b935abc9f47c4c998fce98c61bd4de/s4pi%20Wrappers/MeshChunks/GEOM.cs#L119

anonhostpi commented 2 months ago

I've updated the Kaitai. I've split it into 3 sections, since the UV stitch list and the skin controller are complex types on their own: