flavray / avro-rs

Avro client library implementation in Rust
MIT License
169 stars 95 forks source link

Error deserializing avro schemas serialized with Java app #186

Closed albertoortiz closed 3 years ago

albertoortiz commented 3 years ago
rust 1.49.0 

avro-rs = { version="0.13.0", features=["snappy"]}

I can't deserialize any message , always get an error, like **GetAvroSchemaFromMap**. Before was getting Magical header error, what I solved adding this to my message: let body_header = [[b'O', b'b', b'j', 1u8].to_vec(), body].concat(); The messages are fine using a Java avro deserializer, but not sure where is the problem with rust.

schema:

  let raw_schema = r#"
        {
          "namespace": "com.tremorvideo.dmpproducer.models",
          "type": "record",
          "name": "USyncMessage",
          "fields": [
            { "type": ["string","null"], "name": "region" },
            { "type": "boolean", "name": "incremental" },
            { "type": [ "null", { "type": "record", "name": "UserMapping", "fields": [
              { "type": "long", "name": "actionTime" },
              { "type": "string", "name": "provider" },
              { "type": "string", "name": "userIds" },
              { "type": ["string","null"], "name": "uuid" }
            ]}], "name": "userMapping", "default": null },
            { "type": [ "null", { "type": "record", "name": "DMPData", "fields": [
              { "type": "string", "name": "userId" },
              { "type": ["string","null"], "name": "deviceId" },
              { "type": "long", "name": "actionTime" },
              { "type": "string", "name": "provider" },
              { "type": ["string","null"], "name": "partner" },
              { "type": "string", "name": "segments" }
            ]}], "name": "dmpData", "default": null }
          ]
        }
        "#;

reading:

let reader_schema = Schema::parse_str(raw_schema).unwrap();
let reader =
                  Reader::with_schema(&reader_schema, &body_header[..]);

I always get an error.

Thank you

arnaudpoullet-dkt commented 3 years ago

Hi,

I'm sorry to see you didn't get an answer on this. I am having the same problem myself, did you find a solution in the end ?

albertoortiz commented 3 years ago

I ended using protobuf in the producer side. Seems like the avro schema is expected with the message, but I won't do that , another option is to use another lib that provide the schema, it's a similar dependency from crates.io, I don't want to spend more time testing another one.

Using protobuf is pretty simple.