Add Fluentd support

[owentl]

Add support for Fluentd output. This will send messages to Fluentd with a tag of o365.

Also cleaned up a few random comments

[ddbnl]

Awesome, extra interfaces are always welcome so I'll have a look and try to integrate this PR this weekend. I've rewritten the config parser to be less of a PITA and deprecated the command line switches so there'll be some merging to do, and the final commit might differ in those respects from the PR.

Thanks for sharing 👍

[owentl]

Makes sense on the re-write. The way I am using the script though, hate to see the command line switches go away. I created a docker container that I am running via a k8s cronjob. So having the command line args is nice b/c I can use ENVs for that. Still can do the same with the config file, but more work :)

[ddbnl]

That makes sense, I hadn't thought of it that way. Config file is (usually) baked into the container whereas switches can often be managed independent of them without building a new image. The biggest reason I removed them is because with all the new interfaces the amount of cmd line args was growing rapidly, but if there's a good use case for them I shouldn't be lazy and just update them.

I'll paste them back in from an old version while merging your PR and put in the new ones necessary.

[owentl]

You know, thinking about it, a ConfigMap is easy enough to do too. To be honest, config map is a WHOLE not easier than maintaining the CLI args! The worst part was coming up with a scheme for Fluentd b/c F was already used for file..haha

[ddbnl]

Yeah that's the issue I was having, only so many letters in the alphabet lol.

I'm merging your PR but I'm missing a dependency for "from fluent import sender". Have I correctly assumed you used "fluent-logger" from PiP here?

[ddbnl]

Your PR has been merged, thanks again for the effort. I changed the way the interface worked slightly, because I felt it didn't need caching. The reason the SQL interface uses caching for example is because sending a log to SQL a million times is super slow, compared to sending a million logs at once. If fluentd is anything like Graylog you can just fire logs at it without any issue. If I was wrong about this let me know and we can put caching back in.

If time allows could you test the new executable with fluentd, as I'm not familiar with it? Copy the fluentd output section of the fluentd config example first, as Tenant Name has to be put in the config file now. Here's the new version:

Linux: Win:

[owentl]

Thanks for merging this in!

You are right, it doesn't need the caching, it will behave like graylog!

You are correct on fluent-logger, I could have sworn I put that in the requirements.txt, sorry about that!

I am sorry, I am on a Mac and don't have a windows/linux machine that I can test the executable on. For what I am using the script for, I am actually not using the executable at all. I am just using the python directly in a python container.

[ddbnl]

No worries, might give me an opportunity to check out fluentd sometime :)

I'll close this PR. If you have any questions/suggestions/requests in the future feel free to let me know. As a heads up, I'm about to release 2.0 in the coming days, which (optionally) replaces the collector engine with one written in Rust. In my own tests it has been at least 10x faster (bottenecked by my laptop NIC, so could be even faster). It's still a python script, as the rust engine is converted to a CPython module.

If faster run times are important to you it's worth checking it out (and any feedback would be useful to me).