Connectors now should support returning an Array

[KurtStam]

This is a...

[*] Feature request
[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Documentation issue or request

Description

With https://github.com/syndesisio/syndesis/pull/3895 addressed, connectors like SQL, Twitter, etc should now support returning an array of entities, on top of the split functionality we have in place currently. This should be a setting when selecting the action.

For example SQL using "SELECT * FROM todo", currently sends a message for each row matched (and in a flow you only get the first match), should have the option to return an array with allow rows. So if this is set the datashapeOutput body will be of type array.

[lburgazzoli]

We get the first match only because of the sql post processor explicit discard all the rows but the first The Split EIP should also work for intermediate steps if needed (of course in addition to allow returning an array)

[KurtStam]

@lburgazzoli what's the Split EIP? We don't ship a split Step last time I checked. BTW my concern is about the definition of the DataShape.

[lburgazzoli]

The Split EIP is what it is used under the hoods as example from the sql actions that have "From" as pattern so the resulting collection is split into multiple exchanges, see https://github.com/syndesisio/syndesis/blob/master/app/connector/sql/src/main/resources/META-INF/syndesis/connector/sql.json#L148.

The data shape do not need to change to use the Split EIP as DataMapper will receive a new exchange for each matching row with a body that matches the DataShape.

Btw, I've nothing against returning an array, my comment was just to let you know that there is also the possibility to automatically split a collection if needed.

[KurtStam]

@lburgazzoli yes well not completely 'automatically' right, because the outputShape is different when using array. So I think we need to add a setting on the Connection configuration screen so that the user can set what output shape is expected:

1. single entities & split, or
2. array and no split.

[lburgazzoli]

I'm not sure if the data shape need to be different as imho it should describe the type of elements of the collection, not the collection itself.

We could add:

• a tag to the action telling that a collection can be returned
• an attribute/tag to the step to configure if split has to be applied or not

The DataMapper UI can then use the step attribute to determine if the datashape is part or not of a collection.

Of course that's only my humble opinion :)

[zregvart]

And to be honest #3895 only addresses the data shapes used with API connectors (client/provider). But the principle of wrapping the collection so it's not a top level element of the data shape's schema is applicable for other connectors.

And yeah, we should either let the users specify or do something magical behind the scenes. And with the something magical I think that we can look at the cardinality of the step/connection before and after the insertion point of the mapper and add split/aggregate as needed.

[lburgazzoli]

This relates to #3856

[christophd]

So I have found following connectors that are returning a list of objects as outcome:

• Servicenow
  • Action: servicenow-action-retrieve-record
  • DataShape: json-schema
• Kudu
  • Action: kudu-scan-connector
  • DataShape: json-schema
• OData
  • Action: odata-read-connector
  • DataShape: json-schema
• SQL
  • Action: sql-connector
  • DataShape: json-schema
  • Action: sql-start-connector
  • DataShape: json-schema
• Google Sheets
  • Action: sheets-get-values-connector
  • DataShape: json-schema

These connectors were analyzed but are not able to return a collection basically because of how the underlying Camel component is working:

• AWS-S3
  • Connector has been using implicit split before but underlying Camel component is using a polling consumer providing the collection element by element as a stream of exchanges
• Twitter
  • By default creates a list of exchanges out of results that are processed one after another

[lburgazzoli]

I think salesforce is missing here

[christophd]

@lburgazzoli I had a look at the salesforce operations used in Syndesis connector and could not find any that applies for potential collection result