[Bug]: Serialization of nested hkArray is wrong

SARDONYX-sard commented 2 months ago

Version

0.1.2

Reproducible errors

Who found the bug: ry14

I found the cause: there was a problem with the serialization order of hkArray< hkaAnnotationTrackAnnotation>.

The hkArray in the nested Class member was inadvertently serialized in-place. In this case, the arrays of the members of the Class within the array (annotationTracks) (the Class with time & text) must be serialized after the annotationTracks is written.

I don't think this problem will be solved without a major change in the way arrays are serialized, but it's a bit of a challenge. It will take some time.

 00000340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
           <-     trackName     -> <-    annotations

 00000350: 02 00 00 00 02 00 00 80 00 00 00 00 00 00 00 00  ................
                                -> <-    trackName      ->

# ----------------------- Wrong hkxc output -------------------------------
-00000360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
-00000370: b0 aa 2a 3f 00 00 00 00 00 00 00 00 00 00 00 00  ..*?............
-00000380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
-00000390: 46 6f 6f 74 4c 65 66 74 00 00 00 00 00 00 00 00  FootLeft........
-000003a0: 46 6f 6f 74 52 69 67 68 74 00 00 00 00 00 00 00  FootRight.......
-000003b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
-000003c0: 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 00  ................
# ----------------------- Wrong hkxc output end  ---------------------------

+00000360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80  ................
           <-               annotations                 ->

log

2024-09-10T07:23:02.278012Z TRACE serde_hkx\src\bytes\ser\sub_ser\structs.rs:162: padding: 4 -> current position: 0x340
2024-09-10T07:23:02.278019Z TRACE serde_hkx\src\bytes\ser\sub_ser\structs.rs:59: local_fixup of annotationTracks: src(264/abs: 0x2b8), dst(400/abs: 0x340)
2024-09-10T07:23:02.278024Z DEBUG serde_hkx\src\bytes\ser\mod.rs:508: serialize struct hkaAnnotationTrack(A class within a field.)
2024-09-10T07:23:02.278027Z TRACE serde_hkx\src\bytes\ser\sub_ser\structs.rs:226: Serialize `StringPtr` field(0x340): trackName("")
2024-09-10T07:23:02.278030Z TRACE serde_hkx\src\bytes\ser\sub_ser\structs.rs:235: Add local_fixup_iter_src: 400()
2024-09-10T07:23:02.278033Z TRACE serde_hkx\src\bytes\ser\sub_ser\structs.rs:311: serialize Array field(0x348): annotations
2024-09-10T07:23:02.278039Z TRACE serde_hkx\src\bytes\ser\sub_ser\structs.rs:59: local_fixup of annotations: src(408/abs: 0x348), dst(432/abs: 0x360)
2024-09-10T07:23:02.278044Z DEBUG serde_hkx\src\bytes\ser\mod.rs:508: serialize struct hkaAnnotationTrackAnnotation(A class within a field.)

test.zip

Expected behavior

The classes inside the array must be serialized later. (It is not clear how this ordering can be resolved.)

2024-09-10T07:15:27.931652Z TRACE serde_hkx\src\bytes\de\map.rs:76: fields: ["trackName", "annotations"]
2024-09-10T07:15:27.931656Z TRACE serde_hkx\src\bytes\de\map.rs:80: deserialize 0th field(0x340): trackName
2024-09-10T07:15:27.931659Z DEBUG serde_hkx\src\bytes\de\mod.rs:316: local_src: 400/abs(0x340)
2024-09-10T07:15:27.931662Z DEBUG serde_hkx\src\bytes\de\mod.rs:334: local_dst: 6208/abs(0x19f0)
2024-09-10T07:15:27.931667Z TRACE serde_hkx\src\bytes\de\map.rs:80: deserialize 1th field(0x348): annotations
2024-09-10T07:15:27.931672Z DEBUG serde_hkx\src\bytes\de\mod.rs:647: current_position: relative(0x198)/abs(0x348)
2024-09-10T07:15:27.931675Z DEBUG serde_hkx\src\bytes\de\mod.rs:657: in_struct array_size: 2
2024-09-10T07:15:27.931677Z DEBUG serde_hkx\src\bytes\de\mod.rs:316: local_src: 408/abs(0x348)
2024-09-10T07:15:27.931680Z DEBUG serde_hkx\src\bytes\de\mod.rs:334: local_dst: 6224/abs(0x1a00)

Relevant log output

No response

SARDONYX-sard commented 2 months ago

~~If there is a solution, it is dynamic dispatching using TaskQueue, but there is concern about speed reduction.~~

Currently don't know how due to ownership and lifetime errors and non object safety due to use of generics in Serialize trait.

SARDONYX-sard commented 2 months ago

We are trying a method that repeatedly jumps to and back from the destination of the data to which the pointer points. If this can be done, the global_fixup ordering problem will be solved at the same time.

SARDONYX-sard commented 2 months ago

It was a hell of a task, but it was done.

SARDONYX-sard / serde-hkx