birkenfeld / serde-pickle

Rust (de)serialization for the Python pickle format.
Apache License 2.0
185 stars 27 forks source link

Isssue with deserializing nested Enums/Stucts #28

Closed zachcp closed 2 months ago

zachcp commented 2 months ago

Hi @birkenfeld ,

Thank you for this great crate. I wanted to use it to deserialise and process Pymol PSE files. I've made quite a bit of progress building a nice set of structs to serialize molecules but now I want to add support for other types of objects stored in these files.

So I'm hoping you might be able to suggest a way forward - either directly or stategically. Thank you! ( I can post the full code somewhere if that would be helpful.

Struct-Based Parsing

// works great
#[derive(Debug, Deserialize, Serialize)]
struct ToplevelStruct {
    data: PyObjectMolecule,  //<- this is a deeply nested struct
}

Enum + Struct

// fails no matter what I do 
#[derive(Debug, Deserialize, Serialize)]
#[serde(untagged)]
enum PymolSessionObjectData {
    Molecue(PyObjectMolecule),
}

#[derive(Debug, Deserialize, Serialize)]
struct ToplevelStruct {
    data: PymolSessionObjectData,  //<- Use the Enum to match. I thought this would be straightforward.....
}
zachcp commented 2 months ago

Repo:

Insight/pointers very much appreciated.

zachcp commented 2 months ago

I ended up writing some custom Deserialization functions. I'm still not really famillar with Rust so it feels like I'm banging around until I get the thing to work .... but its working and passing tests.