Enable the pipeline editor to pull meta information from wrapped assets

[romeokienzler]

Enable the pipeline editor to pull meta information / possible configuration parameters from wrapped assets like notebooks or python scrips by reading a common header/common cell/field. This way, the notebook can tell the pipeline editor which additional configuration fields are expected and render them in a separate tab.

[lresende]

This is something we have been talking about and we can consider this in two different ways:

• One approach is to simplify "bring your own components" to Elyra which is what we propose in #500, that would meet all your requirements as the component.yaml would define all the parameters, etc and the runtime would take care of adding the component to the canvas palette and request the user to add/fill parameters.

• The other way of thinking about this is to enable similar support for parameters as Papermill and use notebook tags to define these parameters. In this case, the UI would allow users to enter the parameter values and we would propagate/integrate with the papermill execution. Having said that, for python scripts, that would probably be built in from the beginning.

The main difference between the two is the implementation cost, where option 1 is much more flexible it would take a little longer to be available in Elyra as it requires several new enhancements to the runtime and UI. Having said that, the second option should be doable with much less effort.

[romeokienzler]

I think in the long run it might even make sense to have both options available. Option 1 is much more sound and flexible but also requires more effort on developer's side.

Option 2 only requires skills to create and execute a NB/python script and define/read parameter values.

If option 2 could be implemented without too much hassle I'd prefer Option 2 first (as I'm in contact with clients already interested to try it out).

Option 1 nevertheless would be a very welcome addition in the long run.

So if I could choose, if you can deliver Option 1 AND 2, this would be great but I'd go for option 2 first as it already solves MY problem.

[romeokienzler]

@lresende #385 is already a very good start - this way text field style input is supported - I suggest to expand this a bit. We could extend the functionality of the parser to read the comment field behind the definition of the environment variable to support suggested content and dropdowns/checkboxes

The following line will cause "DATA_SOURCE=jdbc://user:pass@host:port/db" to be added to the properties list data_source = os.environ['DATA_SOURCE'] #jdbc://user:pass@host:port/db

The following line will cause a drop down field (> 3 options) or radio button list (<= 3 options) to be rendered for WRITE_STRATEGY containing update, insert and upsert as options write_strategy = os.environ['WRITE_STRATEGY'] #update|insert|upsert

The following line will cause a selectable drop down field (> 3 options) or check box list (<= 3 options) to be rendered for RELATIONS_TO_READ containing user, account and transaction as options

relations_to_read = os.environ['RELATIONS_TO_READ'] #user&account&transaction

@marthacryan please have a look at this issue and give it a thought how this relates to your latest change of moving from a text area for the input variables to a list

@akchinSTC fyi

[romeokienzler]

In #500 a palette.json is mentioned - the suggested ENV definition could - as an intermediate step - render a palette.json and the rendering is taken from the palette.json - that way the UI is already prepared for palette.json

if there is a JSON string defined in the notebook and assigned to a variable called "palette" - this is used to render the UI and the ENVs are ignored