ElManchacho / FilebeatToCloud

An easy way to add a Filebeat configuration to your system
0 stars 0 forks source link

Take in account that gathered Logs can have multiple "stable" format #13

Closed ElManchacho closed 1 year ago

ElManchacho commented 1 year ago

Create a script that will take in account multiple logs attributes configurations and that will match the lines with the correct attributes :

Example :

header1;header2;header3 value1.1;value2.1;value3.1 value3.2 value1.3;value2.3;value3.3 value1.4;value3.4 value3.5 value3.6 value3.7 value1.8;value3.8 value1.9;value3.9 value1.10;value2.10;value3.10

We can see 3 different logs formats here : not every field is filled for every line.

That would be a problem with a simple Filebeat mapping configuration : the lines without the exact number of mapped fields mapped would be ignored.

value1.3;value2.3;value3.3 --> OK

value1.4;value3.4 --> 1 field missing (2nd column)

value3.5 --> 2 fields missing (2nd and 3rd columns)

ElManchacho commented 1 year ago

fieldsConfiguration.py