ArroyoSystems / arroyo

Distributed stream processing engine in Rust
https://arroyo.dev
Apache License 2.0
3.67k stars 206 forks source link

Add support for configuring timestamp serialization format #585

Closed mwylde closed 5 months ago

mwylde commented 6 months ago

This re-adds support for controlling how timestamps are serialized in JSON via the json.timestamp_format option on tables. The heavy-lifting is done by this change in our arrow-json fork: https://github.com/ArroyoSystems/arrow-rs/pull/3.

I've also exposed the ability to set explicit_nulls in the serializer and have enabled that, getting us back to the behavior in 0.9.

For JSON deserialization, the format of the data and the timestamp precision controls how timestamp fields are handled—e.g., a number in a TIMESTAMP(3) field will be interpreted as unix millis, while a string in any timestamp precision will be interpreted as a RFC3339 value.

I've also validated that alternate precisions are correctly supported when used as event_time fields.