[Feature request] Logformat

[theonlydoo]

Hi,

Since maltrail is a great tool but doesn't have any 'proactive' feature, I try to peer it with various softwares like fail2ban or ossec-hids. There is an issue there, since log format is csv by default, and the filename format is yyyy-mm-dd.log it's quite difficult to handle for several reasons :

• log format could be like main.log and after rotation yyyy-mm-dd.log (and why not yyyy-mm-dd.log.gz )
• log format could be customizable to match syslog format, so it's easier to parse with standards tools

[stamparm]

1) you can always access the current date's log with:

cat /var/log/maltrail/$(date +"%Y-%m-%d").log

2) in some point in time i'll introduce .gz. Also, I am not very keen for log rotation here because logs here are not so big in size and they should be left as kind of "evidence" for potential future analysis. I know that in our organization we care about maltrail logs for future use (hence, no log rotation) 3) as of syslog format. I don't want to introduce unnecessary content there. This is a plain CSV and it is readable by any CSV reader 4) please take a look into: https://github.com/stamparm/maltrail/issues/30#issuecomment-175091570. This looks like a good way how to deal with this whole issue

[theonlydoo]

Hi,

Although 1, 2 and 4 are not so important, I think that 3 is quite important.

Let me explain : With OSSEC (or fail2ban), you have to use pattern matching, when you parse logs. To do that, you have to use a consistant log format. On this link, you can check what it implies in terms of regex.

Now, on a more "edge" case, like a nmap, log format looks like this. You can see that victims IP are 1.2.3.4 and attacker ip is 5.5.5.5 in the logfile. It becomes hard to extract information from a "usual" pattern in this case (and some others).

That was why I suggested to add a "log format option" in the config file.

[theonlydoo]

<decoder name="ossec-maltrail">
    <prematch>(\S+\s+\S+\s+\S+\s+)</prematch>
    <regex offset="after_prematch">(\d+\p\d+\p\d+\p\d+)</regex> 
    <order>srcip</order>
</decoder>

Here is the best decoder that i've managed to create so far, I'll keep running tests

[MikhailKasimov]

@theonlydoo Hello!

I'll keep running tests <-- How them are going?

What is done:

1. ```Option FAIL2BAN_REGEX...```` (See README.md)
2. https://github.com/stamparm/maltrail/issues/13251
3. LOGSTASH_SERVER option for sending logs in JSON-format.

[theonlydoo]

Hey @MikhailKasimov thank's for your reach out; but I gave up maltrail a long time ago, I'd be glad to reach out if I ever use it again. I'll run tests on this issue :-)

Cheers.

[MikhailKasimov]

@theonlydoo Welcome back! There're really many new interesting changes have arrived to Maltrail. Any questions, feedbacks --welcome any time!

https://github.com/stamparm/maltrail/blob/master/CHANGELOG

[stamparm]

@theonlydoo can you please write down what was bothering you with it? maybe we could use your feedback to make it better

[theonlydoo]

@stamparm thank's for asking but idk if you noticed, this issue is dating back from 2016 :-) a lifetime ago!

I have no recollection whatsoever of what I might have bumped on using maltrail

[Thanatermesis]

Same here, I need to use a static "maltrail.log" file in order to make my fail2ban being able to ban automatically attacker IPs

logrotate will automatically (with an extra conf) take care about zipping the files and rotating them

[stamparm]

@Thanatermesis have you tried accessing the Maltrail server with path /fail2ban? for example, <maltrail_server_ip>:8338/fail2ban?

---

Reference: https://github.com/stamparm/maltrail/blob/master/README.md#server