matanolabs / matano

Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS
https://matano.dev
Apache License 2.0
1.46k stars 99 forks source link

Zeek log source does not accept TSV format #177

Open hilt86 opened 1 year ago

hilt86 commented 1 year ago

Zeek writes logs using TSV / ASCII format by default. These logs are gzipped (by default) by the Zeek log rotation script so most folks will end up uploading gzipped TSV logs by default.

Expected bahaviour : zeek tsv files are un-gzipped and added to the matano data lake

Actual behaviour : the transformer function fails with

INFO transformer: {
    "bytes_processed": 1607225,
    "error": false,
    "failing_log_sources": null,
    "log_sources": [
        "zeek"
    ],
    "matano_log": true,
    "rows_written": 0,
    "service": "transformer",
    "sidelined_lines_count": null,
    "sidelined_log_sources": null,
    "time": 165,
    "type": "matano_service_log"
}