search

illuin-tech / data-pipeline

Toolkit for describing data transformation pipelines by compositing simple reusable components.
MIT License
5 stars 0 forks source link

API: allow constructing pipeliner builders from pre-existing pipeline builders #25

Open eledhwen opened 5 months ago

eledhwen commented 5 months ago

The is currently no way to derive a pipeline builder from an existing pipeline builder instance.

This type of feature is useful for situations such as having a Spring bean for a "base" pipeline builder instance, which other beans would fork from, tweak, and then build into their respective pipeline bean. We can already have something similar by creating "factory" functions instead, but in this case we need to explicitly wire all required arguments (which can introduce discrepancies).

The general idea is along the lines of:

var newBuilder = Pipeline.of(existingBuilder);
nitintecg commented 5 months ago

@eledhwen I would like to contribute on this issue but having one doubt below : Are steps and sinks fields necessary only for creating a pipeline builder from an existing builder? Thanks

eledhwen commented 5 months ago

Hi there.

Thank you for chiming in! I'm not sure I understand the question, do you use the library in some capacity?

It might be appropriate to familiarize yourself with what it does (and how) before diving in potential contributions.

This is a "good first issue" as long as one has some grasp on what is actually being attempted.