Tech Extension

[rhuss]

Provide a possibility for an expert user to create an archive which can be uploaded to Syndesis via the UI. This archive contains the definitions for one or more Steps which are offered to the citizen user when creating an integration. The steps themselves consist of some kind of mini-routes (or route fragments), but look like a single unit from the outside.

The following tasks are required:

• Tech Extension Metadata specifying the steps with there meta-data (name, icons, ..)
• Data Shape Info of the steps (what comes in, what goes out)
• Unit testable
• Management: iPaaS UI -> Settings - Tech Extensions (List, Import, View/Inspect, Delete)
• Java DSL
• XML DSL
• Integration Editor shows all steps available when a step can be selected

Beside this, there is an IDE integration required for helping in creating tech extensions.

[lburgazzoli]

Wondering if we could create a sort of RouteConnector on camel side that just forward to a route using either direct , seda, etc and then we can use an eventually simplified Syndesis Connector + Action as extension meta data.

So we can have:

• Extension -> Syndesis Connector
• Exposed Route -> Syndesis Action

Then a step would look like:

- kind: endpoint
  uri: extension:action

[dongniwang]

Tech Extension Metadata specifying the steps with there meta-data (name, icons, ..)

I thought Technical Extensions would show up as a step. We don't have icons for step in the UI right now. See screenshot below.

Management: iPaaS UI -> Settings - Tech Extensions (List, Import, View/Inspect, Delete)

• List: Is it a list view of the available technical extensions?
• Import: Is it uploading the technical extension? So upload a file?
• View/Inspect: What do users want to see and what actions they need to take here?

@rhuss thank you!

[rhuss]

@dongniwang yes, forget the icons. I thought there were some for the steps. But might be interesting to see whether a step comes from a tech extension (e.g. as a right align, small label in italics ?) wdyt ?

List: Is it a list view of the available technical extensions?

Yes, this would be the list of all uploaded technical extensions (which in turn contain one or more steps).

Import: Is it uploading the technical extension? So upload a file?

Yes, its a file generated by the expert developer.

View/Inspect: What do users want to see and what actions they need to take here?

I would expect the user to see :

• name of the tech extension
• list of all included steps, in a format similar to the list shown when selecting a step
• other meta-data which we don't know yet (could be author, creation date, version, ....)

---

Another question which comes to mind is whether it should be possible to enable / disable tech extensions ? The steps for a disabled tech extension simply would not show up when creating new integrations, but would not affect existing integrations. wdyt, @kcbabo ?

[rhuss]

@lburgazzoli the idea sound good. However for this story we dont want to tackle connectors, but directly steps. For "Custom connectors" there is an extra story.

So we can have:

Extension -> Syndesis Connector Exposed Route -> Syndesis Action

The problem with Syndesis connectors is that you have to create "Syndesis Connection" first before you can use a connector in an integration. But what we want here is to directly use it for the steps without any other configuration, so that we can't use a connector.

We could treat the steps within an extension as camel-connector directly (which actually already specifies the "Syndesis Action". still confusing, I know). The good thing would be that we already have a way to declare meta-data like the parameters required for this step. These would then show up in a form when the step is selected.

But yeah, that probably boils down exactly to your idea to re-use the meta-data machinery of the camel connectors.

I think this trait is worth to follow up. We should quickly think about an example and try to realize a PoC here.

[rhuss]

@lburgazzoli while we talked in the chat about dependencies: The important thing is that a tech-extension can be deployed as a single jar, as self-contained as possible. The core camel dependencies should not be included of course (but the meta data should specify to which camel versions the extension is compatible). Other dependencies should be included, fat-jar like.

The boundary is fuzzy and I wonder whether this is possible at all ?

[lburgazzoli]

@rhuss

We do not need to create any new custom connector, we have to create a simple "forwarding" connector we can then extend with actions so on syndesis ui a users would see the extension as any other conenction (twitter, salesforce, etc) which has one or more actions (mapped to routes).

Not sure it it is clear :)

[lburgazzoli]

@rhuss

• dependencies: so an extension should list all the camel deps like using gav and shade everything else which opens the dire to different version of the same class being on classpath, i.e. differet version of guava :)

• extensions I think there is a little bit of confusion between what connector means in syndesis and camel :) but what I meant is that to make an extension one do not need to create any new camel connector but it has to create a json that syndesis can see like twitter, salesforce & co.

  Something like:

  [
  {
    "kind": "extension",
    "data": {
      "id": "extension-name",
      "name": "Extension Name",
      "description": "an extension",
      "icon": "fa-puzzle-piece",
      "configuredProperties": {
      },
      "properties": {
      },
      "actions": [
        {
          "id": "com.mycompany:enrich:latest",
          "name": "Enrich Data",
          "description": "Creates a post to Facebook",
          "camelConnectorGAV": "io.syndesis:direct-connector:@syndesis-connectors.version@",
          "camelConnectorPrefix": "direct-connector",
          "definition": {
            "inputDataShape": {
               "kind" : "any"
            },
            "outputDataShape": {
               "kind" : "any"
            },
            "propertyDefinitionSteps": [{
              "name": "target",
              "description": "the target",
              "display": false,
              "properties": {
                "target": {
                  "kind": "path",
                  "displayName": "path",
                  "required": true,
                  "type": "string",
                  "tags":[],
                  "secret": false,
                  "componentProperty": false
                }
              }
            }]
          }
        }
      ]
    }
  }
  ]

  Under the hood when the project is generated, syndesis will create an alias for a direct connector that matches the integration name so application secrets should include something like:

  direct-conenctor.configurations.${extension-name}.some-property = some-value

  so then the following code is possible:

  - kind: endpoint
  uri: extension-name:action-id

  The action do not need to have any configurable parameter (we could add some additional attributes to the action parameters so only that are can be modified can be shown on the ui). In this case the action can have a directId parameter not modifiable so the step would look like:

  - kind: endpoint
  uri: extension-name:directId

  Make any sense ?

[lburgazzoli]

@rhuss

We could also create a syndesis camel component named i.e. syndesis-extension/step so extension routes would have an entry like

    from("syndesis-extension:action")
       ...

or:

    from("step:action")
       ...

This would help the tooling and a maven plugin that would automatically generates the required meta-data as the entry points to the extension can be easily distinguished.

[lburgazzoli]

Note:

We can leverage Spring Boot's PropertiesLauncher to load extension (packages with additional dependencies) from an arbitrary path, references:

• https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/loader/PropertiesLauncher.html
• https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#executable-jar-property-launcher-features

I wronte a simple example:

• https://github.com/lburgazzoli/spring-boot-extension-example

[chirino]

I think we need to update this epic with a specific user story from @kcbabo to clarify the type of extension he want to tackle first. This story should drive the demo that @lburgazzoli gives to us to consider this epic complete.

[gashcrumb]

For the config I'd recommend having the actual settings the user enters in a separate configuredProperties attribute as name-value pairs, as this would be consistent with how steps and actions are configured in Syndesis.

[lburgazzoli]

@kcbabo do you mind updating this epic with a user story ?

[kcbabo]

@lburgazzoli yep - will do today

[kcbabo]

Created #151 and #152 to track citizen integrator and developer user stories. I would be happy to work with you on creating a meaningful example scenario as well. Probably best for us to collaborate on that vs. me inventing an example out of thin air. :-)

[rhuss]

@lburgazzoli just a small reminder that we started the last week of this sprint. As I have to update the plans for next week, do you have a rough idea what is left over for Sprint 20 (only high-level, more than in next week's sprint planing) ?

[dongniwang]

Integration Editor shows all steps available when a step can be selected

I'll be working on some initial research/design draft for invoking technical extension step in integration editor. Please let me know if you have any information about what kind of configurations users would see/interact when invoking technical extension steps.

@lburgazzoli

[lburgazzoli]

@dongniwang this is hard to define as options definition is left in the hand of the developer so I think we can just show them as a simple list.