search

brimdata / zed

A novel data lake based on super-structured data
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.35k stars 67 forks source link

Schema versioning (building block) #2639

Open philrz opened 3 years ago

philrz commented 3 years ago

Some readers of the Zed platform materials have noted that there's not yet mention of formal schema versioning concepts, whereas other data lake systems tend to cover this heavily. Due to the flexible & self-describing nature of the Zed type system, we suspect that use cases that rely heavily on schema versioning may not be a perfect fit for Zed in the near-term. However, @mccanne has some thoughts about a building block we could add now in the ZSON spec to start covering this, such as describing a scheme for versioning the type names (foo.1, foo.2, etc.) and explain that this is a building block that allows the wider schema versioning problem to be solved above the ZSON layer.

Likely an MVP1 target.

mccanne commented 2 years ago

Elevating priority of this given recent discussion with Team Bears.