search

AmadeusITGroup / workflow-controller

Kubernetes workflow controller
Apache License 2.0
24 stars 15 forks source link

workflow-controller should allow exporting the dag. #33

Open chirag04 opened 6 years ago

chirag04 commented 6 years ago

eg: given a workflow with dependent steps, workflow-controller should allow statically querying the execution-plan / dag. This is much more like how you can statically analyses a DB query.

thoughts?

sdminonne commented 6 years ago

@chirag04 mind explain a little bit more this issue? Do we need to export the DAG (feasible with an external tool) I'm not sure I understand everything about "statically analysis a DB query"... But I'm really not a DB guy. :)

chirag04 commented 6 years ago

This could very well be exposed as an external tool but i believe we already have this tool in our code here.

@sdminonne imagine steps like these defined in the workflow spec:

- step A
      .....
- step B
      ....
      depends on: step A
- step C
     ....
- step D
     ...
     depends on step B and step C.

if should be possible to query what the DAG will look like even before running the workflow. in this case it would be something like:

A   -----------> B    ------->  D
C ---------------------------^

I think we already compute that graph, just that we should expose that computation as an api / util. example output could be something like a json

{
    a: [ 'b' ],
    b: [ 'd' ],
    c: [ 'd' ],
    d: [],
}

where each node stores the edges it has to other nodes. This can be used to display the DAG on UI etc.

sdminonne commented 6 years ago

OK. Thanks. But I don't think workflow-controller should expose an API to "export" a graphviz-like representation of the workflowSpec. What if an external pod would watch the same CustomResource expose that API? @clamoriniere1A @EricMountain-1A ideas?

sdminonne commented 6 years ago

Obviously the pod I'm talking would use the same code used by workflow-controller.

clamoriniere1A commented 6 years ago

@chirag04 also it can be done in the future workflow-controller-ui that we have in our nice to have todo list with @sdminonne

sdminonne commented 6 years ago

yeah but if they already have something we may expose something... Or @chirag04 may start a small project to do this.

chirag04 commented 6 years ago

I don't think workflow-controller should expose an API to "export" a graphviz-like representation of the workflowSpec

why not? i would imagine we already have the code for that here. we could def use that for workflow-controller-ui as well.

sdminonne commented 6 years ago

'cause I do want all the threads and CPU horse-power for controller threadiness. Exposing an API like that it's a simple task "but" with a REST endpoint that I would keep out of the controller.

chirag04 commented 6 years ago

sure. didn't mean to say we have to have a REST endpoint for this. could be a simple stand-alone utility/script outside workflow-controller as well. I just thought we already have that code here, it would be cool to make it a reusable for static analysis.

that said, i haven't looked at the codebase yet. i will check and see if we can make it a simple script here or outside workflow-controller.

sdminonne commented 6 years ago

@chirag04 to expose what I called a graphviz-like representation you just need a simple that does something like 

$ kubectl get workflows -o json > workflows.json

That is going to give you a json file that you can transform with the tool you like (jq?) or python or whatever. If you want to do something more reactive you may want to observe workflows or implement a simple informer that watch workflows and change REST endpoint accordingly to the latest changes (if any). As soon you have a proto or similar les us know and we may eventually have a look (if you need/think/are ok).

sdminonne commented 6 years ago

Keep this issue open to track this discussion which is interesting for various topics.