search

brimdata / zui

Zui is a powerful desktop application for exploring and working with data. The official front-end to the Zed lake.
https://www.brimdata.io/download/
Other
1.8k stars 132 forks source link

Ingest ZNG-format Zeek logs #693

Closed philrz closed 4 years ago

philrz commented 4 years ago

In the original vision for opening Zeek logs as captured in https://github.com/brimsec/brim/issues/566, it was expected that ZNG would be among the accepted formats. At the moment (Brim commit f837314 talking to zqd tagged v0.12.0) attempting to load such logs into Brim gives the "Unknown file types" error, so it seems we still have work left to do here.

Like we did for Zeek TSV/JSON, we could provide a hyperlink from any ZNG reference, in this case to https://github.com/brimsec/zq/tree/master/zng/docs.

henridf commented 4 years ago

It sounds like this issue would require extending the Brim app's autodetection to recognize zng formatted logs? If so an alternate option might be to add a "output type" flag to zq (that just runs zq's detection mechanism and writes the detected format to stdout) and have Brim run zq with that flag that way to detect the format of input files. The zq work for this would be super easy, and this way Brim wouldn't have to re-implement another detection scheme. Just a 2c idea.

alfred-landrum commented 4 years ago

@henridf : thank you, our plan from discussions today is to make this very simple: Brim will only try to identify pcaps. If it doesn't identify a file as a pcap, it will assume it's some type of log, and send it via the log ingest api (including the json typings), and let zqd figure out what format it is.

jameskerr commented 4 years ago

@mason-fish The functionality of this has been fixed in this PR: https://github.com/brimsec/brim/pull/716 Let's just make this ticket about the UI (add zng to the list of supported formats with a link to the zng docs.)

philrz commented 4 years ago

Verified in Brim commit 7e0f417 talking to zqd commit 43d474d. In the video below, I drag the ZNG and TZNG logs from the zq-sample-data into separate tabs of the Brim app and they both import fine.

Verify.mp4.zip

Thanks @mason-fish!