idaholab / Malcolm

Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files), Zeek logs and Suricata alerts.
https://idaholab.github.io/Malcolm/
Other
356 stars 59 forks source link

replace kbn_sankey_vis with vega or transform #147

Closed mmguero closed 11 months ago

mmguero commented 1 year ago

EDIT: see my latest comments below for a working version of this, I think now it's just to implement and tweak

Malcolm's sankey visualizations are using my "osd_sankey_vis" fork of the uniberg/kbn_sankey_vis. However, opensearch compatibility is not something that's upstream and it's difficult to keep my (somewhat hackneyed) fork up-to-date.

I think this can be done, however, using Vega. See:

mmguero commented 1 year ago

Here's an example of a "working" sankey, but I'm not sure how to get the table/composite aggregation to order the size by bucket/node size.

action.hjson.txt

mmguero commented 1 year ago

lguillaud/osd_transform_vis is also an option.

mmguero commented 12 months ago

This blog post could be very helpful: https://blog.davidvassallo.me/2023/09/08/adding-opensearch-dashboards-kibana-filters-to-vega-visuals/

https://gist.github.com/dvas0004/8f0cc265b15e4726490c65c6947016d5

mmguero commented 12 months ago

this actually seems to work pretty well:

sankey.vega.txt

mmguero commented 11 months ago

This is a work-in-progress of a three-level sankey, don't have it quite right but it's close. I may just split it into two for now. The only one that used a three-tier graph was the actions and results, and actually thinking about it having it split into 2 might be better since otherwise you have to have action:* AND result:* (which is not the case for all protocols, so some are being left out).

sankey.vega_three.txt

mmguero commented 11 months ago

Done. Things that could still be improved slightly:

But for the most part it's even better than before, because now selection results in applying a filter.