flavray / avro-rs

Avro client library implementation in Rust
MIT License
169 stars 95 forks source link

How to access Avro data easily? #197

Closed ultrabug closed 2 years ago

ultrabug commented 2 years ago

Hi,

Sorry if my question sounds naive but I'm trying to use this crate to manipulate Avro data stored into (Confluent) Kafka with schema registry using schema_registry_converter.

I asked there first but I have the feeling that my current method to access the rdkafka BorrowedMessage key/values decoded to a types::Value Record trait is not the best. So I'm allowing myself to request some guidance since the examples are not helping.

I'm confused about the fact that a debug print of the types:Value Record trait shows a list of key/value tuples with Unions or String or Arrays etc, that look to have been resolved using the Avro schema but I fail to access them in what seems to me to be a convenient way (like object.property).

Example code (my questioning in comments):

use avro_rs::types::Value;
use avro_rs::from_value;
use schema_registry_converter::async_impl::easy_avro::EasyAvroDecoder;

[...]

let decoder = EasyAvroDecoder::new(sr_settings);

match consumer.recv().await {
            Err(e) => warn!("Kafka error: {}", e),
            Ok(m) => {

                let decoded = decoder.decode(m.payload()).await.unwrap().value;
                // this prints a Record but 'decoded' is a avro_rs::types::Value
                // ex : Record([("oid", String("asdfsdf")), ("uid", String("7873b372-497b-4953-8832-aaaa")), ("fd", Union(String("7873b372-497b-4953-8832-dddd"))), ("td", Union(Null)), ("cd", Union(String("7873b372-497b-4953-8832-sdsdsdsdsd"))), ("platform", String("room")), ("tag_id", Int(6310)), ("action_id", Int(6))])
                // println!("decoded = {:?}", decoded);

                // should I really do that to get the vec! of a the Value::Record?
                let record = match decoded {
                    Value::Record(v) => v,
                    _ => vec![],
                };

                // that looks aweful and cumbersome, I must be missing a proper way to access the data
                for (k, v) in record {
                    match k.as_str() {
                        "oid" => println!("oid = {:?}", from_value::<String>(&v)),
                        "uid" => println!("uid = {:?}", from_value::<String>(&v)),
                        "tag_id" => println!("tag_id = {:?}", from_value::<i32>(&v)),
                        "action_id" => println!("action_id = {:?}", from_value::<i8>(&v)),
                        _ => (),
                    };
                }
            }
        };

I tried using "the Avro way" and "the serde way" from the documentation, when Deserializing using the serde way and a Struct, I get deserialization errors on missing fields tho my understanding was that it should silently ignore them. So I tried focusing on the Avro way and the best I could get was this ugly (to me) way ot manually accessible fields that should have already been resolved...

I'm sure something elegant is obvious to you guys, pardon my Rust newbie question :innocent:

Thanks

ultrabug commented 2 years ago

Closing, see #200