search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.75k stars 8.15k forks source link

[Automatic Import] Support CSV formatted logs #194342

Open ebeahan opened 5 hours ago

ebeahan commented 5 hours ago

Summary

Add CSV formatted logs as a supported log type for Automatic Import.

Background

To format CSV into JSON documents the ingest pipeline provides a CSV processor. The postgresql integration package contains an example of a csv ingest processor.

CSV (or other delimited) logs may or may not contain the header line with field names corresponding to the fields in the rest of the file. Having the header information makes it simpler to create the key/value mapping. If field names are not provided, more processing to determine those names will be necessary.

Example log format

1,2019/11/23 00:44:44,01234567890,AUTHENTICATION,login,2561,2019/11/23 00:44:44,vsys1,fe80::4e7:1ab2:f6aa:82fa,user,normalize-user,object,auth-policy,12345,auth-id,vendor,log-action,server-profile,description,client-type,event-type,10,20,action-flag,0,0,0,0,vsys-name,device-name,vsys-id,auth-protocol,uuid,2021-11-23T01:03:05.498-08:00,src-category,src-profile,src-model,src-vendor,src-os-family,src-os-version,src-hostname,aa:aa:aa:aa:aa:aa,region,,"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",session-id
elasticmachine commented 5 hours ago

Pinging @elastic/security-scalability (Team:Security-Scalability)