search

twitter-archive / kestrel

simple, distributed message queue system (inactive)
http://twitter.github.io/kestrel
Other
2.77k stars 312 forks source link

journal file size on disk grows #16

Closed Suhail closed 14 years ago

Suhail commented 14 years ago

Why does the journal file size increase on disk when you're just reading items from kestrel and acking them which would remove them from the queue? That seems...odd. Shouldn't it get smaller if anything?

robey commented 14 years ago

the journal is a journal of operations, so removing an item and acking an item are operations that are journaled. those operations should only take a few bytes each, though. eventually the journal is compacted so that it only contains live items.

andrewclegg commented 11 years ago

I know this issue is long closed, but I can't find an answer anywhere else.

We have queues where there are hundreds of gigabytes of old journal files for queues that currently have items=0. Obviously this takes up a lot of disk space, but it also means Kestrel takes ages to start up, as it replays all those old operations.

How can we force Kestrel to compact them?

robey commented 11 years ago

That sounds like a bug. It should erase old journal files as they roll over. Can you file a new issue?