Show conference logs in a more convenient way

[rexim]

Please get rid of that frame! It's ugly and inconvenient!

[ForNeVeR]

Well, there's a plan to reuse horta-web API here. We should discuss how do we plan to reuse it - whether we want to replicate all logs from the horta database or lazily replicate it on demand or simply send browsers to horta-web itself.

[ForNeVeR]

Okay, currently the plan is to update Yog-Sothoth archivarius bot to be netcore-compatible. To do that, we'll need netcore-compatible XMPP library, and I'm actively working on port of Jabber-Net library.

The task will have a positive impact on both netcore ecosystem and cross-platform XMPP stack availability, so that's a very good thing.

At the same time, netcore stack is still evolving, so I hope we'll be able to benefit from upcoming netstandard 2.0 stuff that should make the porting process easier.

[ForNeVeR]

Alright, the current idea is to:

• [x] pack the beta shit from SharpXMPP and upload to NuGet (https://github.com/vitalyster/SharpXMPP/issues/7)
• [ ] implement a simple logger robot (not Yog-Sothoth but a simpler one) and run it as part of ctor web service
• [ ] push the logs into a PostgreSQL database (no more NoSQL, sorry)
• [ ] implement a simple scriptless page that will show the current logs from the database

[ForNeVeR]

To reiterate:

• We have an archive of old logs collected during 2009-08-22 — 2016-12-09, uploaded to ctor and backed up on my computer. These were collected by DeadBot (an old freqbot instance hosted by 0xd34df00d). I remember the time zone on the DeadBot's computer to be very off occasionally.
• We have an archive from my Miranda IM in form of a bunch of text files, which may be useful in restoring some information not available in DeadBot's logs. These files are currently unavailable to me, but should arrive in a month or so.
• Telegram logs of codingteam are available since 2016-10-25.
• Since 2017-01-25, we have Telegram to XMPP integration available, that mostly works and syncs the messages (though the timestamps are obviously different from XMPP). There were two different integration bots (older one using chat-linker, and a newer one using Emulsion).
• We also have a Matrix room that's not connected to the other ones, but we have plans to do so. And it's also a part of our history, and thus should be a part of our logs.
• All the previous time, we were relying on jabber.ru storing out chat logs, but somewhere between 2022-02-24 and today these were taken down completely.

So, we have several different chat networks sometimes interlinked, and several different sources of information (sometimes possibly contradicting each other!) are available for certain log periods. And currently we have no working log solution, and no history of the recent events.

This should be fixed.

The idea of reconciliation of several different log sources and creating a combined log archive is pretty old (I remember us discussing it in the great old days of c_plus_plus@conference.jabber.ru, when we were discussing integration of Letnan-Ferry – remember that fella? – and DeadBot), and we have some traces of it in our various IM-related projects:

• cthulhu-bot from 2009 is able to read the Miranda text log format, though it's difficult for me to reember the exact purpose of that process
• there's styx-miranda, my project to synchronize logs online between several IM client instances
• horta-hell's is also able to read logs from Miranda's format via its MarkovPlugin
• we have aforementioned horta-web, a service that was reusing horta-hell's database if I remember correctly, to present the logs in a web page
• we also have Yog-Sothoth (my, my, what a name!), a .NET based chat logging service
• we have a branch named 27-archivarius in this very repository, where I was trying to introduce yet another simple XMPP-based solution

Most of the stuff is discontinued, and/or XMPP-focused, but we need something bigger here. Some challenges:

• so far, there's no single "best" source that contains everything (even in the scope of one chat network), so we have to combine the information from several sources
• the information sources contain a lot of overlapping data, and sometimes even contradict each other, and even themselves (since the timestamps differ a lot between the sources, and sometimes the same message may be repeated several times in old XMPP logs)
• XMPP and Telegram are sometimes connected and sometimes they aren't, which creates some problems in log reconciliation between the two. We should properly recognize the synchronized messages sent via one of our bots.
• Telegram logs aren't available via the normal bot API, and only available via MTProto.

So, I believe that we'll need to create a separate application (or just a service in this repository? TBD) that

• will allow the admin to upload the logs in one of the supported formats, including some form of Telegram messages dump (or perform a quick one-shot Telegram message import online)
• will try to automatically deduplicate the available messages across different sources, and allow the admin to do it manually if required
• will also implement a Telegram bot API to catch and log any new messages online
• I don't think we need an XMPP bot right now, since Emulsion is more or less reliable these days
• I believe it's also a good idea to try to recover our XMPP logs between 2016-12 and 2017-02 in case they contain anything important
• Matrix should also be supported in the future, including the ability to import message history, or receive the logs online
• the data should be imported in raw format (e.g. deduplication shouldn't alter the original data)
• there should be a web API and a web UI that allows to see various data attributes (source, amount/types of sources confirmed a particular event in case of deduplication) and filter on these attributes