kamu-data / kamu-engine-spark

Spark application for streaming transactional transformations between datasets
Apache License 2.0
6 stars 0 forks source link

Switch back to uint64 offset and u8 op column types #12

Open sergiimk opened 8 months ago

sergiimk commented 8 months ago

We currently violate ODF spec that specifies unsigned types for offset and op columns.

This was done due to very old parquet version in Spark 3.2, but recent upgrade to Spark 3.5 allows us to finally rectify this.

This task is to: