search

matrix-org / dendrite

Dendrite is a second-generation Matrix homeserver written in Go!
https://matrix-org.github.io/dendrite/
Apache License 2.0
5.67k stars 664 forks source link

reduce naffka_messages table size #1847

Closed grinapo closed 2 years ago

grinapo commented 3 years ago

Description:

This is a one-man low-traffic dendrite.

dendrite_naffka=# select count(*) from naffka_messages ;
  count  
---------
 1414524

public | naffka_messages | table | dendrite | 1960 MB |

dendrite_naffka=# select min(message_timestamp_ns),max(message_timestamp_ns) from naffka_messages ;
         min         |         max         
---------------------+---------------------
 1612649629637155169 | 1619594406723549138
neilalexander commented 3 years ago

It's quite possible that we will be replacing naffka/kafka with something entirely different, so this problem will probably go away.

If you want to free up the space by hand in the meantime, it should be safe to drop everything from this table and then truncate all the rows from each of the _partition_offsets tables. You should stop Dendrite while doing this.

grinapo commented 3 years ago
  1. naffka_topics shall be TRUNCATEd as well
  2. as you mentioned on matrix one shall not forget to TRUNCATE all _partition_tables. (I didn't but maybe someone will read this later)
neilalexander commented 3 years ago

We're evaluating a switch to NATS at the moment (as per #1866) — this problem will be fixed if we do, as we'll be able to set a proper retention policy for each topic.

kegsay commented 2 years ago

We're using NATS now.