search

brimdata / zui

Zui is a powerful desktop application for exploring and working with data. The official front-end to the Zed lake.
https://www.brimdata.io/download/
Other
1.8k stars 132 forks source link

Ensure "Pivot to logs" modifications will parse and return intended results #1420

Open philrz opened 3 years ago

philrz commented 3 years ago

In a team discussion on 2021-02-02, we discussed an issue raised in the community from a user that was trying to modify the Z behind the Activity Overview query to include the Suricata alerts (Slack thread). Their initial attempt was:

fuse | count() by _path, event_type | sort -r

This produced a table that looked correct, but the right-click "Pivot to logs" operation then created Z that did not target the correct rows, e.g. _path="files" event_type=null because the original non-fused data lacks an event_type field in the files records and hence this boolean and fails to match any records.

I tried my hand at coming up with one which also did not work:

put _path=event_type | count() by _path | sort -r

In that case the "Pivot to logs" on the alerts line generates the Z _path="alert" which will return no results because the put only added the _path field as a prerequisite for running the aggregation, but the Suricata records in the original data lack the _path field and so the Z will find nothing.

We've seen other cases where the Z generated by a "Pivot to logs" will actually fail with a parsing error.

@mccanne made the point that this could be a problem/solution in the domain of Zealot. He seemed to think it could be made smart enough to examine a flowgraph and determine the possible legal pivots. At minimum this would hopefully allow us to "gray out" the "Pivot to logs" option if we know it will fail to parse or will not return the intended results. Perhaps it will also all pivots to be successful on sophisticated Z that for which pivots don't work today.

(Side note: The Z enhancement described at https://github.com/brimsec/zq/issues/2066 is yet another example of functionality that, if made possible in Z, one would want to be able to pivot from.)

jameskerr commented 3 years ago

A simple improvement would be to run the generated "pivot-to-logs" Z scripts through the parser to see if is at least valid or not. If it throws an error, then gray out the menu item.