Open progval opened 5 months ago
Option 2 was my plan here; the network sync layer already provides an eventually consistent view of what needs to be stored, so streaming that independently into long-term storage shouldn't pose any issues. Queries can then be directed to any online history-capable node.
The only time this doesn't work is when adding a new history node, or if one has been offline longer than the lifetime of messages in the synchronised state. Either of those will require a stream from an existing node to get to a point where the global state can start from, but I think doing this as an occasional batch process will be easier than trying to do it live.
Currently, all servers serve all the history. This allows
sable_ircd
to directly answer IRCv3 ChatHistory requests from clients. However, this means memory size puts a hard limit on the retention period. Libera.Chat gets about 1 million messages a day and assuming an average 1 day retention period and 1-10kB per message, this means 1-10GB of memory use already.The idea (from @spb) to fix this is to add dedicated servers for long-term storage. They would use an off-the-shelf database, like PostgreSQL, to store the history past what fits in
sable_ircd
servers' memory; andsable_ircd
would query it over the RPC when needed (ie. ChatHistory requests for old messages, IRCv3 Search, ...).Additionally, we want this history database to be replicated, for durability. As a bonus, we may be able to get high-available and even load balancing. There are two ways to design this:
Option 2 is (IMO) way simpler because we don't need any coordination at all. Option 1 has the advantage of possibly being able to share
sable_network
's design, if that's what we settle on in #119.