Compute Step: add toInt, toDouble and filter(list, expression) functions

[eolivelli]

Summary:

• add new built-in functions to the compute step

• add README

• fn:toDouble: converts a value to a double

• fn:toInt: converts a value to an integer

• fn:filter(list, expression): filter a list type and keep only some values

The main usage is:

• perform a "query" step and assign the results to a value.results field in the message
• run compute step to filter some of the values and create a new list

"fields":[
     {
       "name": "value.filteredResults", 
       "expression" : "fn:filter(value.results,'fn:toDouble(record.similarity) > 0.7')"
     }
]

Please note that it is important to use "fn:toDouble" because the "query" step aways return string values and you have to convert the value to "double" in order to perform the evaluation correctly.

[eolivelli]

@cbornet @aymkhalil @nicoloboschi the solution here is a bit hacky because it wasn't supported to return an ARRAY type.

it works well for the schemaless json case (I added unit tests) and it works also for AVRO in case the list is a list<map<string,string>> (the results of the query step). But it won't work with ARRAys of other types

[cbornet]

Do we really need functions toInt and toDouble ? For the other operations we rely on type coercion. Maybe that would also work for fn:filter and make it more user friendly ?

[cbornet]

Do we really need functions toInt and toDouble ? For the other operations we rely on type coercion. Maybe that would also work for fn:filter and make it more user friendly ?

@eolivelli @aymkhalil

[aymkhalil]

Yes. Coercion should do. Could be as generic as coerceToNumber.