search

wso2 / streaming-integrator

A stream processing runtime that allows connecting any streaming data source to any destination and act on it
Apache License 2.0
113 stars 50 forks source link

Custom mapping flow does not engage if all the custom mappings are relevant to the transport attributes. #239

Closed janithcmw closed 2 years ago

janithcmw commented 2 years ago

Description: If the @attrubutes contains only the transport level attributes, the CSV mapping will not go through the custom mapping. It will pass through the default mapping[3]. In default mapping, the attributes will grab from the stream definition and because of this, there can be index out of bound issues.

Affected Product Version: WSO2SI-1.1.0 and WSO2SI-Tooling-1.1.0

OS, DB, other environment details and versions: N/A

Steps to reproduce: Define a custom mapper as follows and it will throw an "index out of bound" error. If there is a mismatch in the elements of the row(not equal to 2) which is red by the file source.

@source(type='file', mode='LINE', dir.uri='file:/<location>/source', tailing='false', header.present='false', @map( type='csv', delimiter='|', @attributes(filePath = 'trp:file.path', eof = 'trp:eof'))) define stream FileReaderStream (filePath string, eof string);

Related Issues: N/A

dnwick commented 2 years ago

This is a user error where we have not used correct delimiter. We can close this

dnwick commented 2 years ago

Issue exist when we have less number of values in the csv line than trp attributes , Hence reopening

dnwick commented 2 years ago

As we analyzed further this is not a blocker and we can easily use this by adding one custom attribute along with trp attributes. And to properly fix this gonna introduce api changes hence we are holding this for now.