mtth / avsc

Avro for JavaScript :zap:
MIT License
1.28k stars 148 forks source link

Rust vs JS encoding #375

Closed joshbenaron closed 2 years ago

joshbenaron commented 2 years ago

Hey,

I've noticed that the output of the JS version is different from the Rust one. The Rust one appears to include the JS bytes + lots more bytes. You'll see the JS bytes are a subset of the Rust bytes. Do you know why they differ so greatly? e.g. JS:

Uint8Array(13) [
    2,   8, 110, 97, 109,
  101,  10, 118, 97, 108,
  117, 101,   0
]

Rust:

[
    79,
    98,
    106,
    1,
    4,
    22,
    97,
    118,
    114,
    111,
    46,
    115,
    99,
    104,
    101,
    109,
    97,
    134,
    2,
    123,
    34,
    116,
    121,
    112,
    101,
    34,
    58,
    34,
    97,
    114,
    114,
    97,
    121,
    34,
    44,
    34,
    105,
    116,
    101,
    109,
    115,
    34,
    58,
    123,
    34,
    116,
    121,
    112,
    101,
    34,
    58,
    34,
    114,
    101,
    99,
    111,
    114,
    100,
    34,
    44,
    34,
    110,
    97,
    109,
    101,
    34,
    58,
    34,
    84,
    97,
    103,
    34,
    44,
    34,
    102,
    105,
    101,
    108,
    100,
    115,
    34,
    58,
    91,
    123,
    34,
    110,
    97,
    109,
    101,
    34,
    58,
    34,
    110,
    97,
    109,
    101,
    34,
    44,
    34,
    116,
    121,
    112,
    101,
    34,
    58,
    34,
    115,
    116,
    114,
    105,
    110,
    103,
    34,
    125,
    44,
    123,
    34,
    110,
    97,
    109,
    101,
    34,
    58,
    34,
    118,
    97,
    108,
    117,
    101,
    34,
    44,
    34,
    116,
    121,
    112,
    101,
    34,
    58,
    34,
    115,
    116,
    114,
    105,
    110,
    103,
    34,
    125,
    93,
    125,
    125,
    20,
    97,
    118,
    114,
    111,
    46,
    99,
    111,
    100,
    101,
    99,
    14,
    100,
    101,
    102,
    108,
    97,
    116,
    101,
    0,
    58,
    255,
    214,
    80,
    175,
    13,
    14,
    83,
    207,
    113,
    35,
    60,
    61,
    14,
    252,
    118,
    2,
    50,
    5,
    192,
    161,
    17,
    0,
    0,
    8,
    2,
    64,
    207,
    228,
    185,
    37,
    129,
    166,
    54,
    153,
    159,
    207,
    58,
    44,
    91,
    152,
    103,
    24,
    58,
    255,
    214,
    80,
    175,
    13,
    14,
    83,
    207,
    113,
    35,
    60,
    61,
    14,
    252,
    118,
]

Schema:

{
    "type": "array",
    "items": {
        "type": "record",
        "name": "Tag",
        "fields": [
            { "name": "name", "type": "string" },
            { "name": "value", "type": "string" }
        ]
    }
}

Value:

[
    {
       "name": "name",
       "value": "value"
    }
]
mtth commented 2 years ago

The Rust one is an object container file. It includes the schema among other things. You can read (resp. write) those with avsc's BlockDecoder (resp. BlockEncoder).