With the recent improvements (especially https://github.com/birdiecare/connect-smts/pull/15) we can carry messages with an extremely large range of shapes without creating dozen of thousands of schemas. The rationale being that Confluent Registry becomes painfully slow to accept new schemas after ~1k schemas on a particular subject. While #15 allows to drastically reduce the number of schemas, it starts its union-ing process from scratch each time it restarts.
This feature allows us to set starting points for the schemas used for each of the JSON fields. It means that we can have a very low number of schemas in the registries (i.e. dozens maximum) while keeping the flexibility of allowing the underlying JSON shapes to change over time, without having to manual change these schemas as they are "starting points" for optimisation only.
With the recent improvements (especially https://github.com/birdiecare/connect-smts/pull/15) we can carry messages with an extremely large range of shapes without creating dozen of thousands of schemas. The rationale being that Confluent Registry becomes painfully slow to accept new schemas after ~1k schemas on a particular subject. While #15 allows to drastically reduce the number of schemas, it starts its union-ing process from scratch each time it restarts.
This feature allows us to set starting points for the schemas used for each of the JSON fields. It means that we can have a very low number of schemas in the registries (i.e. dozens maximum) while keeping the flexibility of allowing the underlying JSON shapes to change over time, without having to manual change these schemas as they are "starting points" for optimisation only.