Add a way to split on a specific field

[mmuzikar]

See also https://issues.jboss.org/browse/ENTESB-11488

This is a...

[x] Feature request
[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Documentation issue or request

Description

This might be only usable for OpenAPI connectors, but from what I've tried so far, there is no way to iterate over a collection in OpenAPI response, because there is some other information (such as status and response type), and split currently only takes the "root" of the message. Which causes for the exchange to be broken into pieces that don't make sense to work with. Example of what I am saying: There should be a way for either specifying that the field body should be split, or a way to modify current body, so I could do body = body.body and then the split step should work.

[asmigala]

The same issue makes it impossible to use a collection POSTed to an API provider integration.

[zregvart]

Just to clarify. The WebHook, API provider and custom API connectors use a unified data shape. That data shape includes the request details (path or query parameters, headers) and the request/response body. For instance:

{
  "parameters": {
    "a": "value",
    "b": "other value"
  },
  "body": {
    "id": "123"
  }
}

The split step we have works on top level collections, this was a design decision, so a nested one like:

{
  "parameters": {
    "a": "value",
    "b": "other value"
  },
  "body": [
    {"id": "123"},
    {"id": "321"}
  ]
}

Will not be eligible for the split step.

@gaughan do you agree that this is a p0.

If it's still critical, I propose that we mark split/aggregate as tech preview in this release and work on this for the next release.

As an alternative we could make the split step aware of this Syndesis convention and consider if the body is a collection, though I'm not sure if that'll cascade and cause other issues.

[christophd]

@zregvart I think this Syndesis convention currently only applies for API provider. Other connectors such as Webhook directly provide the message body as data shape.

However split will not be able to operate on these data shapes provided by Webhook for instance as it would require some auto conversion from json array String to a list of json beans. This auto conversion to a list of json beans is done at the moment by some connectors explicitly providing a collection (such as SQL, Kudu, OData, ServiceNow, GoogleSheets).

So I think the split could be enhanced to deal with those kind of collection data shapes where the body is a json array string.

Secondly the Syndesis convention used by API provider could also be added to the split logic. We could easily do that by adding some meta data information to the data shape that marks the shape to use this convention. Or add a split expression to the data shape that tells the split what attribute of the shape should be split by default.

[heiko-braun]

p0 is for bugs, hence removing it here.

[zregvart]

@christophd FYI About the JSON schema for specifying request parameters, used in API Provider, custom API connectors and Webhook.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!

[zregvart]

@tplevko you can add status/never-stale label and the stale bot will leave it alone...

[tplevko]

@zregvart totally forgot about it, thanks!

[heiko-braun]

Migrated to https://issues.jboss.org/browse/ENTESB-11488