[solved] how to compile slsa

[nicopace]

I cloned the repo, and did the steps in the travis file. I got the lognorm binary but none of the programs in tools were compiled. When i read the tools makefile, the lines that build the slsa command were commented, saying it is not compatible with v2. any way of testing it? I have a ton of devices (wireless routers that are part of a 100s devices mesh network) that will be dumping their logs into a rsyslog server that would love to keep an eye on, and being able to categorize its logs would make it much easier. (https://librerouter.org :D ). Thanks!

[davidelang]

I'm not familiar with the slsa command, so I don't know what you are trying to do. The v2 functionality changed a lot of things in ways that are incompatible with v1, so if you have a program that requires the v1 API, you will need to compile the v1 library, not the later versions.

is slsa something ncluded in the liblognorm repo, or is it an external tool that uses liblognorm?

David Lang

On Tue, 21 May 2019, Nicolás Pace wrote:

I cloned the repo, and did the steps in the travis file. I got the lognorm binary but none of the programs in tools were compiled. When i read the tools makefile, the lines that build the slsa command were commented, saying it is not compatible with v2. any way of testing it? I have a ton of devices (wireless routers that are part of a 100s devices mesh network) that will be dumping their logs into a rsyslog server that would love to keep an eye on, and being able to categorize its logs would make it much easier. (https://librerouter.org :D ). Thanks!

[nicopace]

Hi David, thanks for the prompt response. This is slsa: https://github.com/rsyslog/liblognorm/blob/master/tools/slsa.c It seems to be a tool that looks at a set of logs and predicts rules based on its analysis.

[davidelang]

On Wed, 22 May 2019, Nicolás Pace wrote:

Hi David, thanks for the prompt response. This is slsa: https://github.com/rsyslog/liblognorm/blob/master/tools/slsa.c It seems to be a tool that looks at a set of logs and predicts rules based on its analysis.

ahh, sounds useful. It would need a pretty complete re-write for v2 (a lot of the functionality has changed), and if it's complaining about that, it seems that nobody has spent the time to do that yet.

David Lang

[nicopace]

the compile is not complaining.. it is just not compiling it at all. I don't have Makefile skills so don't know how to trigger its compilation. Any guidance would be greatly appreciated!

[nicopace]

I just uncommented the lines in Makefile.am and now it is failing: https://github.com/Pardus-Kurumsal/liblognorm/blob/a0a209d2af0f0c019df61aa8fcd7da78ab18b19f/tools/Makefile.am#L2 Good point to start working on it.

  CC       slsa-slsa.o
slsa.c:50:0: warning: "_GNU_SOURCE" redefined
 #define _GNU_SOURCE

In file included from slsa.c:49:0:
../config.h:116:0: note: this is the location of the previous definition
 # define _GNU_SOURCE 1

slsa.c:60:32: fatal error: json_compatibility.h: No such file or directory
 #include "json_compatibility.h"
                                ^
compilation terminated.

Any guidance would be greatly appreciated!

[nicopace]

in order for it to compile, we need to restore a file that was removed: https://github.com/janmejay/liblognorm/blob/81efc58cde335e6a099f1d313703420e3dbb54cd/src/json_compatibility.h

it already works (doesn't show how to use it, but basically you pipe a log file to it and it returns a set of rules).

[rgerhards]

Be careful - it's an experimental tool never completed...

[nicopace]

Thanks @rgerhards it is a neat experiment, and a good tool to jump start rule writing. All this comes from loosing a raspberry pi with all the rules I have written (~60), and not wanting to write everything from scratch... this certainly helps!

[nicopace]

And it works remarkably well for being experimental... I throwed 1,000,000 lines of logs to it in an i5 mobile processor + SSD, and took ~20secs to process into 2000 rules, were just some of them were too generic or too specific. Now i am going to select those I want, use lognormalizer to filter out those lines that apply to it, and try again. It would be great if the tool could have some threshold controls (like when it thinks it is a good fit or not to join two lines together). Thanks so much!

[nicopace]

And btw, the 1.5months long logfile is 232m lines of logs totalling a gzipped file of 1.5gb. This is the processing outcome of it:

\time -v bash -c 'zcat ./terremotolog.gz | head -10000000 | liblognorm/tools/slsa -p > slsa-rules.rb'
reading: 10000001 - done
subword detection: 113310 - done
squashing: 48074 - done
rule table create: 11494 - done
sorting rule table: 0 - done
rule table print: 11494 - done
    Command being timed: "bash -c zcat ./terremotolog.gz | head -10000000 | liblognorm/tools/slsa -p > slsa-rules.rb"
    User time (seconds): 318.28
    System time (seconds): 10.20
    Percent of CPU this job got: 105%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 5:11.31
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 7544328
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 1887874
    Voluntary context switches: 478680
    Involuntary context switches: 6769
    Swaps: 0
    File system inputs: 0
    File system outputs: 2784
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0