ArroyoSystems / arroyo

Distributed stream processing engine in Rust
https://arroyo.dev
Apache License 2.0
3.67k stars 206 forks source link

Fix regression in kafka producer that panicked on key serialization #574

Closed mwylde closed 6 months ago

mwylde commented 6 months ago

Fixes a regression in kafka serialization that causes run-time panics. The deserializer can only deserialize value schemas (as it has logic to project out the non-timestamp columns). In the Kafka producer, we had logic to also serialize the keys if they were present. That was previously ignored, because there never ends up being a key in the graph produced by SQL. However, when we changed the key representation to be an Option<Vec<usize>>, that is now Some but with an empty schema, causing the key serialization code to run but panic due to not having any columns.