Choose an IRC Logger

[CircleCode]

we made the decision during Hoa Apex'15 to use an IRC logger to keep track of discussions, and have the ability to search through them.

the logger must have these features:

• be light
• be robust
• offer a web IHM
• the web IHM must be searchable
• do some mail obfuscation

  Criterias

These questions are some points that can be used as a basis for our choice. This is not an exhaustive list.

• does the logger obfuscates mails?
• can we have only log features?
• is there a way to have not logged messages (with something like [OFF] this message is not logged)?
• if it is a hosted service
  • Is it open source?
  • can we download an archive of the logs in an exploitable format (in case we want to leave the service)?
  • can we ask for some logs deletion? (imagine we put a password in the chan by mistake)
• if it is a self-hosted service
  • what are the system requirements?
  • what level of administration is required?

    Tasks

• [ ] compare loggers
• [ ] choose one of them

[Jir4]

Maybe it might be interesting to have a notification when something happens during the night, we also have to specify in the topic of the main channel our time zone as we said yesterday

[CircleCode]

I considered several daemons.

I first restricted my search to php daemons, but only a few seem light and robust. Then I extended my search, and here is the candidate I found:

https://github.com/moritz/ilbot

• written in perl
• maintained
• in use on several channels
• 62 :star: on GH

@Pierozi what do you think about the stack required for ilbot, would you consider installing it on our infrastructure or do you see any cons?

we can consider adding logging functionality to https://github.com/hoaproject/Marvirc, but this cannot be achieved in a short time, and we would also have to write the web IHM

the other simple and immediate solution would be ton use colabti logger (http://colabti.org/ilogger2/IrcLogger2.html), but this is not open source (btw, it seems trustable) and we do not have the logs physically.

[Jir4]

@CircleCode i don't really understand why upgrade Marvirc can not be achieved in a short time.

[thehawk970]

I think in the night :p

So Marvirc (or other) can log when we are mentionned and offline. And notify us when we connect , as a little report like an answering machine.

Marvirc: [13h37] @foo : I need ...

@Jir4: +1

[vonglasow]

@Jir4 @camael24 +1 aggreed with improve marvirc. We just need to log each line and it's easy because marvirc already read it the "only" thing to do is to know where to store it and how this will be searchable. For the web IHM a searcher like this we have in hoa-project looks good.

[Jir4]

I dont think we are mentionned when we are offline, users ping someone in the channel.

[CircleCode]

@camael24 the idea is to log everything so that we can search later. for mentions, I think this is not te role of the logger, but rather the role of a bouncer, or another Marvirc plugin. but the only role of the logger must be to… log everything

@Jir4 my assumption was that if this exists, wa can spend time on other features. But if someone wants to write such a feature in Marvirc, one could open an issue in Marvirc Repo (where we can discuss further about it).

@vonglasow the pure logging part is not so hard (it "just" has to store each line, right) but it has to be designed with search, security, mail obfusaction, etc. in mind. Moreover, I am against adding the logs to the actual search engine, because this would "pollute" the site own results.

[vonglasow]

@CircleCode: The goal was not to add it on the actual search engine but take it as source to do another one dedicated to the log. Why do you need obfuscation for this logging ? Everything is public on IRC. We have nothing to hide :d

[Pierozi]

:+1: for Marvirc but like @CircleCode said, only for logger. Like it's lot of data, I think the better solution are to store in text file, one per each day. Easy to read and write by stream and make a dedicated page on hoa-project.net.

[CircleCode]

@vonglasow when you write your mail on IRC, you know that only people present at this time will read it. when this is logged, it means that this is "on the internet" and can be accessed later by anybody. So we need mail obfuscation

[CircleCode]

@Pierozi like always, good point: if we have text files in a "first version", it seems sufficient. Search and some other features can be added later to the website through another issue.

[Hywan]

Marvirc is a good choice but… what about an external service? We do not want to host everything… And if we use Marvirc, we need to write the web interface, the search engine etc. ourselves.

[CircleCode]

@Hywan that was my point when I suggested to use http://colabti.org/ilogger2/IrcLogger2.html (the only requirement is to ask to be logged…!)

[Hywan]

http://colabti.org/ilogger2/faq.html

Where can I download the source code ?

The source code of the bot is currently not available for download.

:-(

privacy

[CircleCode]

@Hywan that was the concern I had (see my comment) … btw, I think they are trustable

[Hywan]

@CircleCode They are not trustable until we can see their code :-].

[Jir4]

@Hywan :+1:

[CircleCode]

@Hywan @Jir4 so the external service is no longer an option. We still have to choose between ilbot (already written, robust, but not fitting our needs at 100%) and adding log functionnality to Marvirc (can be achieved in several steps, but time consuming and it requires someone to work on it).

Thoughts?

[thehawk970]

@CircleCode +1 :clap:

[Hywan]

@CircleCode We can use an external service that we trust because open sourced. Does it exist? We can ask on Twitter too. Make a tweet and the @hoaproject account will RT it.

[CircleCode]

@Hywan I'll be in vacation for 15 days, and do not use twitter as often as I used to. I think it would be better if the tweet comes from hoa account (by mentioning my account)

[Hywan]

https://twitter.com/hoaproject/status/649196830459301888

[CircleCode]

answering here about WildPHP/Wild-IRC-Bot (new version of wildphp.com/simple-irc-logger mentioned by @Cyrano24): I looked at it, but it seems to much for our needs (we only want a logger, and we already have Marvirc for other bots' tasks)

[Grummfy]

https://botbot.me/ => https://github.com/BotBotMe

[Hywan]

@Grummfy w00t, love it! Trusted by Rust. I made a quick audit of the code, looks code. The web interface is really neat and polished, love it.

[Grummfy]

Yep, me too love the interface ;)

[Jir4]

:+1:

[CircleCode]

@Hywan, @Grummfy love it too, seems reliable (and I had no skill to audit the code, but I trust Hywan's judgment) If we choose botbot, I'd just have a few questions for them, like:

• can we use logging only, and no other feature of their bot
• what if we, for example, put a password on the channel? (can we ask for deletion of some parts of the logs?)
• does the logger obfuscates mails?
• is there a way to ignore messages (with something like [OFF] this message is not logged)?

these questions are not blockers, just informations

[Hywan]

@CircleCode

• What other features are you talking about? They just use !m which is an IRC convention to reward someone.
• For the password, I have no idea. Let's try (or ask)?
• Obfuscates emails, let's ask yes.
• Turn it off temporary, let's ask.

They are on Freenode if I mind correctly. I will ask today if nobody does it.

[CircleCode]

@Hywan maybe you can ask'em on freenode to answer here so that we keep an history, and discuss with them here

[CircleCode]

@Hywan by other features, I was speaking about their plugins

[Hywan]

@CircleCode As far as I know, we are not required to install these plugins. I am going to ask.

[CircleCode]

@Hywan if we choose the hosted service, there is no "installation": we just ask'em to make their bot instance join our channel. In this cas, the meaning of my question was: “can they disable plugins on a per channel basis”

[Hywan]

@CircleCode We can just don't use them ;-), but good point.

[CircleCode]

@Hywan

We can just don't use them ;-)

My concern is about having 2 bots on the chan. I want it to be clear that the botbot one is only here for logging so that we can replace it by another one if we decide it later.

This leads me to another question: is there a way to get the logs back (for example in txt files) at any time?

[Hywan]

@CircleCode Please, write all questions in the description comment.

[CircleCode]

@Hywan done :-)

[Grummfy]

for info on #mongodb chan they put comment when you come on the channel saying that everything is logged :)

[LeoVerto]

@Hywan We've recently started using BotBot as our new @metabrainz chat logger and have been pretty satisfied so far.

While it didn't do the actual setup, it seemed to be pretty easy. The bot is mainly developed for running on botbot.me with public account creation so we recently decided to fork the -web part (logger and webinterface) and are planning to change it to work better with our overall theme and also to remove some stuff we don't need.

Funnily, since we also make use of "[off]", I recently implemented that feature (with a configurable prefix) which is currently an open PR on upstream.

BotBot's plugin system is a bit different that something like Hubot's. Messages are saved in a redis queue and sent read-only to plugins. This means that a plugin crashing won't take the whole thing down, but also prevents plugins from removing messages from the queue, which was my initial idea for implementing [off].

All in all we are quite happy with BotBot and hope to be able to push some improvements upstream in the future.

[Hywan]

@LeoVerto Thank you for your feedback!

[Freso]

I came here to write pretty much exactly what @LeoVerto wrote. :) Anyway, as has been mentioned, there's also the possibility of using https://botbot.me as a service to publicly log the channel for you, instead of having to deal with running your own instance. You asked a couple of questions about this earlier, and maybe I can provide reasonable answers:

• Can you enable/disable plugins on a per-channel basis?
  • Short answer: No. Long answer: It is technically possible, but the BotBot.me administrators will not do this, due to the amount of channels they're hosting and how big their backlog is (e.g., channels having applied for logging but not having gotten accepted/rejected yet). It would simply be too much of an additional burden. (This is also why, even once @LeoVerto's [off]-patch makes it into upstream, it likely won't be enabled on the BotBot.me site/service.)
• Will it be possible to get a dump of the logs if moving chat logger in the future?
  • Probably not. They do not state that they will do this anywhere on their site or anywhere else (that I've seen), and due to the administrative burden of doing this (it really wouldn't be that hard, a simple SQL query should be able to dump everything, but it's still something on top of the pile of other things to do), I highly doubt they would do this. And if they will, don't expect it to get done anytime soon from the time you request it.

It should be noted that I'm not in any way affiliated with @BotBotMe / @LincolnLoop, these are merely observations from (mostly) lurking in #lincolnloop for several months. I was involved in picking @BotBotMe as @MetaBrainz' new logger though, and I still stand by that, but the two points above were some of the major considerations that led to us running our own instance of the logger (as @LeoVerto talked about) instead of using the hosted service at BotBot.me.

[Pierozi]

Thanks both @LeoVerto & @Freso for sharing your experience, it's really appreciate.

@BotBotMe are pretty good, write in Go, good choice for this appliance, and light in consumption. Dependences are PostgreSQL and Redis ( :clap: ) both light to install and handle.

[Hywan]

@Pierozi Because, you imagine installing it on our own servers?

[harikt]

Regarding the bot , no one seems mentioning https://github.com/phergie .

[Jir4]

@harikt because we look for a logger and not for a bot

[harikt]

ok, Thanks for clarifying.

[Hywan]

I sent a request to botbot.me:

[Hywan]

We have no “request ticket”, so we cannot follow up the progression. Waiting.

[CircleCode]

@Hywan so you decided a hosted BotBot.me. I agree with this choice (sorry for the late answer, I was on holidays), but I really think that once Marvirc has logger feature ( Hywan/Marvirc#5 ), we will have to switch to it (so that we can "manage" everything). Until this is done, a hosted solyution seems to be the best. Btw, remember botbot offers a lot of functionalities, and can be conflictuous with Marvirc.

[Hywan]

@CircleCode We can just avoid to talk to BotBot.me and everything is going to be fine. I didn't choose anything, it was apparently the main choice :smiley:.