search

IBM / data-prep-kit

Open source project for data preparation of LLM application builders
https://ibm.github.io/data-prep-kit/
Apache License 2.0
171 stars 111 forks source link

pipeline transform #602

Open blublinsky opened 1 week ago

blublinsky commented 1 week ago

Why are these changes needed?

Initial implementation of pipeline transform

Related issue number (if any).

roytman commented 1 week ago

@daw3rd , I looked over this PR and it is OK for me, but leave it for your final verdict.

blublinsky commented 1 week ago

Can we run a pipeline outside of the runtime/launcher? If so, I'd like to see a test or other example of how this is done? If not, we need it.

For running a pipeline outside a runtime we have kfp. We can not pass values in memory outside of runtime

daw3rd commented 1 week ago

Can we run a pipeline outside of the runtime/launcher? If so, I'd like to see a test or other example of how this is done? If not, we need it.

For running a pipeline outside a runtime we have kfp. We can not pass values in memory outside of runtime

This is a problem and does not meet requirements defined in the issue. https://github.com/IBM/data-prep-kit/issues/374#issuecomment-2353447356

blublinsky commented 1 week ago

Can we run a pipeline outside of the runtime/launcher? If so, I'd like to see a test or other example of how this is done? If not, we need it.

For running a pipeline outside a runtime we have kfp. We can not pass values in memory outside of runtime

This is a problem and does not meet requirements defined in the issue. #374 (comment)

The requirement does not specify the ability to run across runtimes. Besides, where is this requirement coming from?