bincode-org / bincode

A binary encoder / decoder implementation in Rust.
MIT License
2.63k stars 265 forks source link

Why does deserializing slightly different structs from the same bytes work? #677

Closed mattgathu closed 10 months ago

mattgathu commented 10 months ago

I have two similar types that I ser/de with bincode:

pub type SstIndex = BTreeMap<Bytes, IndexVal>;

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SstMetadata {
    index: SstIndex,
    seq_num: u64,
}

During a refactor I realized that the following deserialization code does not error out:

rdr.seek(SeekFrom::End(offset))?; // rdr is BufReader<File>
let mut buf = vec![0u8; meta_sz as usize];
rdr.read_exact(&mut buf)?;
let meta: SstMetadata = bincode::deserialize(&buf)?;
let index: SstIndex = bincode::deserialize(&buf)?;

I'm wondering why the SstIndex deserialization works? Is this behaviour documented somewhere? Thanks

EDIT: I went ahead a did a simple reproduction:

use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;

pub type SstIndex = BTreeMap<usize, usize>;

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SstMetadata {
    index: SstIndex,
    seq_num: u64,
}
fn main() {
    let index: SstIndex = (0..100).map(|i| (i, i)).collect();
    let meta = SstMetadata {
        index,
        seq_num: 0u64,
    };
    let buf = bincode::serialize(&meta).expect("ser failed");
    let index: SstIndex = bincode::deserialize(&buf).expect("deser failed");
    println!("index {:?}", index);
}

cargo run produces:

   Compiling serde v1.0.190
   Compiling bincode v1.3.3
   Compiling bintest v0.1.0 (..)
    Finished dev [unoptimized + debuginfo] target(s) in 7.16s
     Running `target/debug/bintest`
index {0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12, 13: 13, 14: 14, 15: 15, 16: 16, 17: 17, 18: 18, 19: 19, 20: 20, 21: 21, 22: 22, 23: 23, 24: 24, 25: 25, 26: 26, 27: 27, 28: 28, 29: 29, 30: 30, 31: 31, 32: 32, 33: 33, 34: 34, 35: 35, 36: 36, 37: 37, 38: 38, 39: 39, 40: 40, 41: 41, 42: 42, 43: 43, 44: 44, 45: 45, 46: 46, 47: 47, 48: 48, 49: 49, 50: 50, 51: 51, 52: 52, 53: 53, 54: 54, 55: 55, 56: 56, 57: 57, 58: 58, 59: 59, 60: 60, 61: 61, 62: 62, 63: 63, 64: 64, 65: 65, 66: 66, 67: 67, 68: 68, 69: 69, 70: 70, 71: 71, 72: 72, 73: 73, 74: 74, 75: 75, 76: 76, 77: 77, 78: 78, 79: 79, 80: 80, 81: 81, 82: 82, 83: 83, 84: 84, 85: 85, 86: 86, 87: 87, 88: 88, 89: 89, 90: 90, 91: 91, 92: 92, 93: 93, 94: 94, 95: 95, 96: 96, 97: 97, 98: 98, 99: 99}
VictorKoenders commented 10 months ago

SstMetadata has more bytes than SstIndex, so you can deseiralize SstIndex just fine. If you want this to be a runtime error, you can set Options::reject_trailing_bytes

For more information on how the bincode format works you can check the spec

mattgathu commented 10 months ago

Thank you.