philrz
commented
3 years ago The fixes in the linked PRs #1842 and #1847 are verified in Brim commit 6ce102f, which points to zqd commit 75263267.
To repro, I start an import of the test pcap https://archive.wrccdc.org/pcaps/2018/wrccdc.2018-03-23.010014000000000.pcap.gz. While it's importing, I cd to $TMPDIR/zqd-pcap-ingest-* and use cat to append the following additional NDJSON events to the eve.json that's being built. They contain the fields alert.bonus and alert.also that are not present in the JSON typing config for Suricata:
{"timestamp":"2020-03-26T11:03:29.986905-0700","flow_id":1186020564515531,"pcap_cnt":7,"event_type":"alert","src_ip":"192.168.5.51","src_port":53262,"dest_ip":"104.28.19.94","dest_port":80,"proto":"TCP","community_id":"1:DgCqAi4bQYb/4BoblZb/Iz//T70=","tx_id":0,"alert":{"action":"allowed","gid":1,"bonus":"LookAtMe!","signature_id":2013028,"rev":5,"signature":"ET POLICY curl User-Agent Outbound","category":"Attempted Information Leak","severity":2,"metadata":{"updated_at":["2020_04_22"],"created_at":["2011_06_14"]}},"app_proto":"http"}
{"timestamp":"2020-03-26T11:03:29.986905-0700","flow_id":1186020564515531,"pcap_cnt":7,"event_type":"alert","src_ip":"192.168.5.51","src_port":53262,"dest_ip":"104.28.19.94","dest_port":80,"proto":"TCP","community_id":"1:DgCqAi4bQYb/4BoblZb/Iz//T70=","tx_id":0,"alert":{"action":"allowed","gid":1,"also":"LookAtMeToo!","signature_id":2013028,"rev":5,"signature":"ET POLICY curl User-Agent Outbound","category":"Attempted Information Leak","severity":2,"metadata":{"updated_at":["2020_04_22"],"created_at":["2011_06_14"]}},"app_proto":"http"}This causes the warning icon to appear:
When clicked, we see mention of the unexpected fields:
As described in the linked PRs, when such an "extra" field appears within a nested record (such as under alert as shown here) they're not made a seamless part of that nested record, but are still visible within the Log Detail view.
In the time since the above changes were made to relax the ingest behavior and produce the warnings, we became aware that the field that community users had encountered that triggered the original symptom was a top-level vlan field. Using the test pcap https://download.netresec.com/pcap/maccdc-2012/maccdc2012_00002.pcap.gz with the Brim commit 6ce102f and zqd commit 75263267 mentioned above, we once again see the warning:
And the unexpected field once again appears "rightward" in the alert record:
Then the additional typing config for this field was added in #1851, and so repeating that import now in Brim commit 77755a1 that points to zqd commit 817b4a0, vlan now appears at a defined location in the alert record.
In conclusion, we're hopeful that the relaxed/warning code may not even need to be exercised at this point since we've squashed the one case we know of that needed it. However, it's there just in case.
Thanks @henridf!
As of the release of the GA combo of zq v0.25.0 and Brim v0.21.0, the JSON typing config for Suricata alert records is rigid such that we only accept the fields we expect to receive from the embedded Suricata. However, soon after release, a Brim community user reported a problem that (based on the error message) appears to be caused by one or more additional fields having appeared in the alert records that were not defined in the typing config:
The proposal is to relax the rigidity such that for unknown fields, we will do the best we can to convert these to appropriate ZNG types. If records with such additional fields should appear, this would prevent them from being automatically shown in columns in Brim after doing ZQL such as
event_type=alert(*). But consensus is that this is an improvement over blocking the read altogether until we add knowledge of the field to the typing config.These additional fields are not expected to be common. We did significant research in advance before creating what we believed to be an adequately comprehensive schema, and we're still waiting to hear from the user about this specific field so we can track down why we missed it. Therefore, it would seem ideal if the implementation were able to warn the user of the unexpected occurrence and nudge them toward telling us about it. My proposal would be a warning like
(*) - Adding
| fusewould address this, though.