birkenfeld / serde-pickle

Rust (de)serialization for the Python pickle format.
Apache License 2.0
188 stars 28 forks source link

Deserializer confused by python2 pickle #1

Closed vorner closed 7 years ago

vorner commented 7 years ago

Let's say I have this python program:

import pickle

data = {
    "answer": 42,
    "hello": "world",
}

with open("data.pickle", "wb") as f:
    pickle.dump(data, f)

And this rust program:

extern crate serde_pickle;

use std::fs::File;
use serde_pickle::{from_reader, Value};

fn main() {
    let data: Value = from_reader(File::open("data.pickle").unwrap()).unwrap();
    println!("{}", data);
}

It works correctly when the python program is run with python3. However, when I run it in python2 (therefore strings are likely byte strings instead of unicode strings), the rust program crashes when deserializing:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Syntax(Structure("invalid type: byte array, expected a hashable value"))', /checkout/src/libcore/result.rs:860

Another interesting thing is, if I use value_from_reader instead of from_reader, it works correctly even with python2-generated pickle. It is a bit strange, as both deserialize into Value.

birkenfeld commented 7 years ago

Thanks for the reports! I'll have a closer look soon.

birkenfeld commented 7 years ago

This was a missing visit_byte_buf in the Value impl. Will be fixed in 0.4.0.