We want to bring back VAST as a plugin for Tenzir—the database plugin.
Notably, this rewrite of the Tenzir's underlying storage engine aims to achieve the following:
import and export should be generalized as publish and subscribe, respectively, that support sharing data between pipelines under a topic. This allows for decoupling of backpressure by splitting one pipeline into multiple, which in turn enables Load on Scroll.
Data transformations should be atomic replacements on the partition level, superseding the partition transformer underlying the rebuild, compaction, aging, and disk-monitor mechanisms on top of a single transform <expr> <pipeline> operator.
Schema explosion should be manageable by merging schemas with matching names and mergeable layouts.
We want to bring back VAST as a plugin for Tenzir—the database plugin.
Notably, this rewrite of the Tenzir's underlying storage engine aims to achieve the following:
importandexportshould be generalized aspublishandsubscribe, respectively, that support sharing data between pipelines under a topic. This allows for decoupling of backpressure by splitting one pipeline into multiple, which in turn enables Load on Scroll.transform <expr> <pipeline>operator.