search

matrix-org / rust-synapse-compress-state

A tool to compress some state in a Synapse instance's database
https://pypi.org/project/synapse-auto-compressor/
Apache License 2.0
142 stars 32 forks source link

Add more examples how to use synapse_auto_compressor #97

Open rubo77 opened 2 years ago

rubo77 commented 2 years ago

the readme only suggests

 synapse_auto_compressor -p postgresql://user:pass@localhost/synapse -c 500 -n 100

This means, that it will only check in 100 rooms and quit as I understand is as a layman.

I got the result after 100 chunks:

 INFO  synapse_auto_compressor::manager] Finished running compressor. Saved 15891 rows. Skipped 4/100 chunks

this seems to have worked.

Then I started again running with -n 10000 now to try to compress my 56GB database.

Is this the best option? or could you provide more informatinon in the readme please?

rubo77 commented 2 years ago

After some hours it said in the end:

INFO  synapse_auto_compressor::manager] Finished running compressor. Saved 150550506 rows. Skipped 381/3307 chunks

WOW! 150 Million rows in 3307*500=1,700,000 Rooms!

unfortunately the database grew from 56 to 59GB in /var/lib/pistgres, although the gzipped sql export of the table state_groups_state shrinked from 5.6GB to 1.5GB. ( I guess a database VACUUM will be needed in the end?

rubo77 commented 2 years ago

After running VACUUM FULL on my database it shrank down to 23GB.

So we should add this too to the Readme: use VACUUM after compressing to really free up the disk-space.

(I thought we could delete the three extra tables in the database: state_compressor_state, state_compressor_progress and state_compressor_total_progress, but they are so small, that it would be not much space to win)

jahway603 commented 1 year ago

+1 for more examples how to use synapse_auto_compressor and its VACUUM feature.