This PR contains a major re-write of Pipeless from Python to Rust. We took advantage of these changes to support many features that were not possible with the previous architecture by re-designing the whole architecture.
There have also been a significant performance increase and an important decrease in resource consumption.
Benefits
Highly parallelized processing. Version 0.x was conditioned to deploy several workers for parallel processing. Version 1.x contains a single type of Pipeless node, running processing in parallel by default.
Removed the need of configuring streams in a YAML file before running Pipeless.
The App class was replaced by the concept of stage. Stages can be easily plugged and re-used as boxes.
Multi-stream processing. Version 0.x supported a single stream at a time. Version 1.x supports any number of streams without significant overhead.
Significant reduction in resources consumption. Version 0.x required a minimum of 3 processes to work, adding an extra process per level of parallelization. Version 1.x works on a single process, and dynamically distributes loads between processes when required.
Multi-language support. The new architecture allows to support several languages for writing hooks. We are starting with Python and Rust.
Significant performance increase. Some tests indicated 5x speed up with 25% of the resources, which computes a 20x performance increase.
Description of the change
This PR contains a major re-write of Pipeless from Python to Rust. We took advantage of these changes to support many features that were not possible with the previous architecture by re-designing the whole architecture. There have also been a significant performance increase and an important decrease in resource consumption.
Benefits
App
class was replaced by the concept ofstage
. Stages can be easily plugged and re-used as boxes.Possible drawbacks
Applicable issues
Additional information