Tech extension metadata format

[rhuss]

• Configuration items
• Versioning
• Declaring dependencies
• ....

[rhuss]

• Spec of dataShapes (i.e. whether a tech extension produces a specific output)

[gashcrumb]

We do have a format in the UI for driving the forms that the advanced filter step currently uses, as do the action configuration page. I'll have to go and write up some proper documentation for it, but the general format is based on work done in hawtio as an example with some changes here and there.

Opened a task for me to remember here -> https://github.com/syndesisio/syndesis-ui/issues/1109

[lburgazzoli]

As the extension behavior is quite similar to a connector, I'd like to propose a similar format for the meta-data definition:

{
  "kind": "extension",
  "data": {
    "id": "${groupId}:${artifactId}",    
    "version": "${version}",
    "name": "Extension Name",
    "description": "Extension Description",
    "icon": "fa-puzzle-piece",
    "tags": [],
    "properties": [],
    "steps": [
      {
        "id": "${groupId}:${artifactId}:${stepId}",
        "name": "Step Name",
        "description": "Step Description",
        "tags": [],
        "definition": {
          "inputDataShape": {
           "kind" : "any"
         },
         "outputDataShape": {
           "kind" : "any"
         },
         "properties": []
       }
     }
   ]
  },
  "dependencies": [
      "mvn:g/a/v",
      "..."
  ]
}

Notes:

• each extension can provide multiple steps
• id and version are derived from the maven project
• dependencies are generated from maven project
• properties can be defined on extension or step level but on UI side they can only be configured globally (like connections), the split is only to make it clear where the properties applies.

[lburgazzoli]

@gashcrumb @nicolaferraro what do you think ?

[gashcrumb]

@lburgazzoli looks good to me, though not sure what you mean in this last bullet point:

properties can be defined on extension or step level but on UI side they can only be configured globally (like connections), the split is only to make it clear where the properties applies.

How would this map to the step configuration UI I wonder...

[nicolaferraro]

Looks similar to connectors API, so I think it's ok. For what I've understood, properties at extension level are not to be supported now in the ui, are they?

Dependencies could theoretically be extracted from the pom bundled inside the META-INF dir of the jar, but in case of complex pom.xml hierarchies it's impossible to determine the correct version server side. So, ok to get the "mvn:g/a/v" from the tooling into the metadata.

Since we're saying extensions are uploaded jars, probably there should be some sort of "uri" for identifying the jar, unless the combination of id+version identify a precise jar file referenced somewhere else. Wdyt?

[lburgazzoli]

@gashcrumb I meant that an extension step should not have options in the step configuration UI so every property has to be configured globally. As properties at extension level are out of scope for the POC let's discuss this in a separate issue, make sense ?

[lburgazzoli]

@nicolaferraro if possible I'd like to keep version+id (gav) as primary reference, the physical location of the file is then a job for the filestore implementation

[lburgazzoli]

Wondering if we need to rename step to something like `extensionPoint~ to avoid further name clashes:

{
  "kind": "extension",
  "data": {
    "id": "${groupId}:${artifactId}",    
    "version": "${version}",
    "name": "Extension Name",
    "description": "Extension Description",
    "icon": "fa-puzzle-piece",
    "tags": [],
    "properties": [],
    "extensionPoints": [
      {
        "id": "${extensionPointId}",
        "name": "Step Name",
        "description": "Step Description",
        "tags": [],
        "definition": {
          "inputDataShape": {
           "kind" : "any"
         },
         "outputDataShape": {
           "kind" : "any"
         },
         "propertyDefinitionSteps": [],
       }
     }
   ]
  },
  "dependencies": [
      "mvn:g/a/v",
      "..."
  ]
}

NOTES:

• An extensionPoint has now a propertyDefinitionSteps that are mapped to Camel's exchange headers

Then when the integration is built a step may looks like:

"steps": [
  {
    "stepKind": "extension",
    "action": null,
    "connection": null,
    "extension": {
        ...,
        "extensionPoint": {
            ...
        },
    },
    "configuredProperties": {
      ...
    },
    "name": null,
    "id": null
  }
]

NOTES:

• extension and extensionPoint property values can be conflated in the step's configuredProperties, it is then a job of the back-end to properly handle them

@nicolaferraro @gashcrumb thoughts ?

[nicolaferraro]

Now it looks good to me!

[lburgazzoli]

After some discussion:

• extension won't have global properties so route-extensions should leverage message headers for conditional processing
• extension steps will be defined as action like for connector, see syndesisio/syndesis-rest#742

[nicolaferraro]

From a first sight at the discussion in #152, I'd do some changes to the extension metadata.

ID The extension must have a logical id, that is given by "groupId:artifactId" when packaged from the IDE. This id will be used to recognize if a extension is a new extension or if it upgrades a existing extension.

Apart from the logical id, there should be an identifier of the uploaded object. The proposal of REST services (#758) for extensions accepts a binary file and assign it a random generated id, that is included into the extension metadata. I'd call this identifier "id" (in the root of the extension object, as per the linked PR) and call the logical one "extensionId".

Status Users will upload extensions, then the extension will be validated and installed. I'd model this with a Status field. For now, the allowed statuses seem to be ("draft", "installed"). Only one extension per "extensionId" can be in status "installed".

Wdyt?

[lburgazzoli]

ok for both do you mind sending a PR to the proposal doc ?

[apupier]

ID The extension must have a logical id, that is given by "groupId:artifactId" when packaged from the IDE. This id will be used to recognize if a extension is a new extension or if it upgrades a existing extension.

Apart from the logical id, there should be an identifier of the uploaded object. The proposal of REST services (#758) for extensions accepts a binary file and assign it a random generated id, that is included into the extension metadata. I'd call this identifier "id" (in the root of the extension object, as per the linked PR) and call the logical one "extensionId".

Why this "extensionId" attribute is needed? it will be available in the pom.xml embedded in the jar, isn't it? Unless we allow to have a logical ID not identical to the artifactId? Like for instance, if we want to support to have several extensions defined in a single jar.

[apupier]

Status Users will upload extensions, then the extension will be validated and installed. I'd model this with a Status field. For now, the allowed statuses seem to be ("draft", "installed"). Only one extension per "extensionId" can be in status "installed".

it seems that there are several attributes which are provided by the runtime (id, data/status, others?), shouldn't we create 2 examples? (and 2 json-schemas) One for the design definition and one for the Runtime definition?

[apupier]

in the proposal doc, currently, there is one attribute which I don't understand very well data/action/descriptor/target Why the example value is direct:${groupId}/${artifactId}/${actionId} ? it will be always like that? if yes, it seems that it can be derived from existing attributes or context.

[apupier]

"dependencies": [ "mvn:g/a/v", "..." ]

Which kind of dependencies will be supported? Maven only? What will be the syntax for the others? Why do you need to declare dependencies if the Archive format is a sprint-boot module (or the thing which will replace it) which is embedding all "non-provided" artefacts?

[lburgazzoli]

@apupier

1. extensionId : it is not limited to be in the form of groupId:artifactId but for projects generated by us that should be the default and the "best practice"
2. jason schema: I will as soon as possible
3. target : is what implements the extension, so you can have a call to a sub route using i.e. direct (but there's no limitation) or you can call a bean or you can implement your own step handler so no it won't be always like the example (will update it as soon as I have something stable)
4. dependencies: an extension should not embed dependencies we ship i.e. if an extension need camel-twitter, then camel-twitter and all its transitive dependencies should not end up in the archive but should be listed in the dependencies section so the project build know what to include

[lburgazzoli]

Updated with the latest changes:

{
  "kind": "extension",
  "id": "assigned-by-syndesis-server (must not be hardcoded in packaged extensions)",
  "data": {
    "extensionId": "${groupId}:${artifactId}",
    "version": "${version}",
    "name": "Extension Name",
    "status": "draft|installed (must not be hardcoded in packaged extensions)",
    "description": "Extension Description",
    "icon": "fa-puzzle-piece",
    "tags": [],
    "action": [
      {
        "id": "${actionId}",
        "name": "Action Name",
        "actionType" : "extension",
        "description": "Action Description",
        "tags": [],
        "definition": {
          "inputDataShape": {
           "kind" : "any"
         },
         "outputDataShape": {
           "kind" : "any"
         },
         "descriptor": {
             "kind": "endpoint|bean|step",
             "entrypoint" : "direct:${groupId}/${artifactId}/${actionId}"
         },
         "propertyDefinitionSteps": []
       }
     }
   ]
  },
  "dependencies": [
      "mvn:g/a/v",
      "..."
  ]
}