redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.09k stars 817 forks source link

Bug in json_path method for numeric fields #1828

Open natenho opened 1 year ago

natenho commented 1 year ago

Expected

root.text_objects = this.json_path("$.body[?(@.type==1)]")

# In:  {"body":[{"type":1,"id":"foo"},{"type":2,"id":"bar"}]}
# Out: {"text_objects":[{"id":"foo","type":1}]}

Reference: https://www.benthos.dev/docs/guides/bloblang/methods/#examples-92

Actual

root.text_objects = this.json_path("$.body[?(@.type==1)]")

# In:  {"body":[{"type":1,"id":"foo"},{"type":2,"id":"bar"}]}
# Out: {"text_objects":[]}

A curious behavior is that when using operators like >, <, >= or <=, the comparison works as expected:

root.text_objects = this.json_path("$.body[?(@.type>1)]")

# In:  {"body":[{"type":1,"id":"foo"},{"type":2,"id":"bar"}]}
# Out: {"text_objects":[{"id":"bar","type":2}]}

So the current workaround for equality is to use >= && <=

root.text_objects = this.json_path("$.body[?(@.type>=1 && @.type<=1)]")

# In:  {"body":[{"type":1,"id":"foo"},{"type":2,"id":"bar"}]}
# Out: {"text_objects":[{"id":"foo","type":1}]}
mihaitodor commented 1 year ago

Hey @natenho, it looks like that's a limitation in the underlying library. I raised an issue about it here: https://github.com/PaesslerAG/jsonpath/issues/35 and I'll wait for a reply from the maintainer(s) before trying to send them a PR.

As a workaround, I'd consider using bloblang for such queries: root."text_objects" = this.body.filter(x -> x.type == 1).