search

brimdata / zui

Zui is a powerful desktop application for exploring and working with data. The official front-end to the Zed lake.
https://www.brimdata.io/download/
Other
1.8k stars 132 forks source link

Allow user to provide a custom JSON typing schema #694

Closed philrz closed 4 years ago

philrz commented 4 years ago

In its current state as of Brim commit f837314, JSON ingest is ready for the "happiest path" of a user that has Zeek JSON logs that conform to the schema we ship that's based on "stock" Zeek v3.1.2. However, as soon as their environment has an additional plugin or any other customization, they're now prone to seeing warnings that may confuse them and/or missing data, which will generate Support inquiries.

To deal with this, I'm planning to write a user-facing article that describes how to use zq to create a modified JSON typing schema. The only problem with this as things currently stand is that there's not a great way for them to put the schema to use. Two possibilities I see:

  1. (As I think we've discussed in the past...) We could require them to use zq to apply typing to their Zeek JSON logs and turn them into ZNG before posting them into Brim. This would require us to finish https://github.com/brimsec/brim/issues/693. In an absolute pinch, I'd be prepared to point to this option in our next release, but it's not very user-friendly.

  2. I could advise them to overwrite the built-in schema with their custom one, e.g. on MacOS, copy it to /Applications/Brim.app/Contents/Resources/app/config/defaultTypes.json. That would give the seamless experience of being able to drag Zeek JSON logs directly into the app, but creates a supportability hazard since now when they upgrade Brim it'll overwrite their custom schema.

Since we're already so close, I find myself wishing we could just have a place where the user could point to an alternate path for the JSON schema. Since JSON ingest is a bit of a "power user" use case, I'd be fine with not investing significant cycles right now to make this a front & center part of the ingest UX. Instead, something simple in the Preferences menu would totally cover us here.

@alfred-landrum and I were discussing this and liked the sound of something like a checkbox that starts out checked that means the default built-in schema would be used. When unchecked, it would activate a way for the user to enter an alternate path to their custom schema. That way I could reference this option from the article so power users could find it, and all the non-JSON users would be blissfully ignorant.

siskojr commented 4 years ago

Proposal for the default built-in schema and selection checkbox. When un-checked, user will be prompted to enter the location of his schema.

image.png

philrz commented 4 years ago

Verified in Brim commit 670d3b8 talking to zqd commit a09d0d6.

I'd made a modified types.json that was intentionally broken, having deleted the http descriptor altogether from the rules and removed a field/type specification from the conn descriptor. In the video below, I choose to use that types.json via the Preferences menu, attempt to import my Zeek JSON logs, and see the expected warnings from the alternate typing config.

Verify.mp4.zip

Thanks @jameskerr!