There are some remaining discrepancies in the ML processor transforms regarding dot notation and JSONPath, that can yield different results when rendering examples on the frontend, vs. the execution on the backend. As such, this can result in successful transformations when testing on the frontend, but failures when executing ingest/search. Tracking known discrepancies below:
[x] ML search request/response transforms only use JSONPath, and the backend JSONPath package does not support dot notation for arrays. To have feature parity, we should automatically convert any dot notation with arrays (e.g., arrays.0) into dot notation with brackets (e.g., arrays[0])., or see if the functionality to automatically generate the keys, can take in some config to automatically convert to brackets by default, or see if there is some external package that can handle all of this dot -> bracket transformation for us altogether.
[x] discrepancies with arrays for single values. Currently, the default JSONPath query command we execute returns an array by default, even if for a discrete/single/definite value. The backend package has a config field called ALWAYS_RETURN_LIST, set to false. We should see if there is a similar config on the frontend side, and/or if we can parse out whether a path is "definite" or not, and do post-processing.
[x] enforce output transforms are always using JSONPath as required per the backend.
[x] improve tooltip wording to indicate the above discrepancies / limitations
[x] keys with dashes in them fail using the nodejs library - for example, given:
{
"field": {
"sub-field": "value"
}
}
and some jsonpath, like $.field.sub-field, will fail to fetch "value". One workaround is to wrap it in brackets with quotes - for example, $.field["sub-field"] will succeed.
There are some remaining discrepancies in the ML processor transforms regarding dot notation and JSONPath, that can yield different results when rendering examples on the frontend, vs. the execution on the backend. As such, this can result in successful transformations when testing on the frontend, but failures when executing ingest/search. Tracking known discrepancies below:
arrays.0) into dot notation with brackets (e.g.,arrays[0])., or see if the functionality to automatically generate the keys, can take in some config to automatically convert to brackets by default, or see if there is some external package that can handle all of this dot -> bracket transformation for us altogether.ALWAYS_RETURN_LIST, set tofalse. We should see if there is a similar config on the frontend side, and/or if we can parse out whether a path is "definite" or not, and do post-processing.and some jsonpath, like
$.field.sub-field, will fail to fetch "value". One workaround is to wrap it in brackets with quotes - for example,$.field["sub-field"]will succeed.