search

adarga-ai / pachyderm-pipeline-controller

A Kubernetes Operator for controlling Pachyderm Pipeline objects.
Apache License 2.0
8 stars 1 forks source link

Support for CRON inputs #1

Open jmandel1027 opened 3 years ago

jmandel1027 commented 3 years ago

Hi @ollystephens, really interesting project! We're just getting setup w/ our own pachyderm cluster and really like what this controller offers. Are there any plans to support the Cron interface? It's something that we've been investigating and could fit our use-case.Additionally are there any plans to support Spout's as well?

https://docs.pachyderm.com/latest/concepts/pipeline-concepts/pipeline/spout/

We're very much impressed w/ how this controller defines pipelines, we have a gitops platform w/ Flux & Helm and would be much more confident in our workflows if it was integrated w/ it rather than the pachctl tool, and see this as a viable solution.

Thanks again!

-Jason

ollystephens commented 3 years ago

Hi Jason, thanks for getting in touch.

We have some updates that we've yet to publish that - amongst other things - support the entire pipeline spec, as we are now autogenerating it from the Pachyderm SDK. So although we don't use the Cron inputs ourselves, it should support them. Same with Spouts (which we do use ourselves).

I'll strive to get these changes pushed out next week, and will leave this issue open until they are.

Olly

jmandel1027 commented 3 years ago

Hey Olly,

Amazing! thanks for the update, looking forward to these changes, we'll definitely be using this controller for our pachyderm cluster here at 1build

Thanks again, -Jason

jmandel1027 commented 3 years ago

Hey Olly, just circling back here. Any progress with those changes?

Thanks again

jmandel1027 commented 3 years ago

Hey Olly, I'm sure you must be slammed but wanted to circle back, I'm just starting to write a CI/CD flow for our pipelines and this would be a big help since we're using Flux heavily to manage our GitOps and see this operator as a great alternative since we have some pipeline specific needs that this would help us with

Thanks again, -Jason

ollystephens commented 3 years ago

So sorry Jason; I've been off work unexpectedly for a few weeks and was hoping one of the team would've picked this up in my absence but wasn't to be. Now that I'm back I'll have the changes pushed in the next couple of days. Again, apologies for the delay! Olly

jmandel1027 commented 3 years ago

no problem totally get it, looking forward to the changes! will they be compat w/ pachyderm 2.x?

jmandel1027 commented 3 years ago

Hey Olly, just wanted to circle back and see where things landed here, curious to see what sort of improvements y'all have made! Also will there be a native dockerfile to support the controller? or will we have to use buildkit for now.

Thanks again, -Jason

jmandel1027 commented 3 years ago

Hey Olly, any movement on things here?

Thanks! -Jason

jmandel1027 commented 3 years ago

@ollystephens Is this project abandoned? If so, that's a shame because using CRDs to manage pachctl resources would be a major win for orgs embracing gitops.

Thanks, -Jason

ollystephens commented 2 years ago

Hi Jason, Really sorry for the lack of movement. It's not acceptable. It's very much not abandoned; just had some internal issues getting in the way of being able to focus on updating it. Please accept my apologies; it will happen real soon now. Olly

jmandel1027 commented 2 years ago

Hey @ollystephens no problem, I totally understand that's great to hear. Looking forward to seeing the next batch of changes!