holoviz / panel

Panel: The powerful data exploration & web app framework for Python
https://panel.holoviz.org
BSD 3-Clause "New" or "Revised" License
4.84k stars 520 forks source link

Pipeline builder widget #1815

Open stuartarchibald opened 4 years ago

stuartarchibald commented 4 years ago

Is your feature request related to a problem? Please describe.

As a compiler engineer working on Numba I often want to be able to see the impact of applying one or more compiler passes, run in a particular order, on some input representation. I leave "input representation" open as seeing both text and dot graph representations of intermediate representations is useful. The obtaining and viewing of the input/output part I can work out and I can see how I could do a "run one selected pass" with existing tools. The problem I cannot work out is how to run multiple passes in a defined order, essentially, how do you let users build a linear pipeline from a set of predefined stages?

Describe the solution you'd like

I'd like a widget where it's possible to select items from a bucket of items and then they appear in a selected "list", I say list because selection order matters. It should be possible to select the same items multiple times and remove items from the selected list. The purpose of having this widget is to be able to let the user define a linear pipeline of operations built from the operations that are available.

Describe alternatives you've considered

I tried a bunch of things from various existing widgets. panel.widgets.CrossSelector can almost do this with the definition_order flag as False, however items from the initial selection are removed when selected and so multiple selection isn't possible. The panel.widgets.MultiSelector does slightly bizarre things that I can't quite figure out if one of the options is repeated too.

Additional context

I think being able to select a pipeline of functions to apply to something is probably useful in general. Lots of things can be described via a pipeline of transform functions, image processing, signal filtering, cleaning data, etc.

Here's a ASCII mockup using some sufficiently realistic compiler pass names:

Start state:

---------------------------------------------------
| Compiler passes available |          | Pipeline |
---------------------------------------------------
| SROA                      |    >>    |          |
| Loop Rotate               |          |          |
| Vectorize                 |    <<    |          |
| DCE                       |          |          |
|                           |          |          |
|                           |          |          |
---------------------------------------------------

User then chooses "I want to run SROA->SROA->Vectorize->SROA->DCE" and does this by selecting the items in the order they want from the "Compiler passes available" column and clicking the ">>" button after each selection.

End state:

----------------------------------------------------
| Compiler passes available |          | Pipeline  |
----------------------------------------------------
| SROA                      |    >>    | SROA      |
| Loop Rotate               |          | SROA      |
| Vectorize                 |    <<    | Vectorize |
| DCE                       |          | SROA      |
|                           |          | DCE       |
|                           |          |           |
----------------------------------------------------

Also, being able to select items in the pipeline to remove with << could be useful, as could just pressing << with nothing selected to pop the last added item in the pipeline.

Thanks for your help. I'm also entirely willing to test/help out if that's useful!

jbednar commented 4 years ago

Sounds very useful, and in fact would address an issue that long predates Panel, dating back to ImaGen, where we also wanted to define arbitrary pipelines of transformations and lists of patterns using a GUI interface. I'd love to see this supported by both CrossSelector and MultiSelector, not as the default behavior but possible to enable with a flag.