search

matrix-org / rageshake

Bug report server
Apache License 2.0
30 stars 12 forks source link

Rageshake support for object storage #60

Open benbz opened 2 years ago

benbz commented 2 years ago

Is your feature request related to a problem? Please describe. Rageshake can take an unbounded amount of storage space. Object storage is generally more elastic than using local disk.

Describe the solution you'd like Rageshake supports storing rageshakes in object storage (S3-compatible API preferably) rather than local disk so that administrators of Rageshake do not need to worry about disk usage

Describe alternatives you've considered Similar to #59, Rageshake could support a maximum total size of stored rageshakes and delete the oldest rageshakes until under the desired maximum store size.

Additional context n/a

gaelgatelement commented 2 years ago

@michaelkaye : Any opinion on this issue + issue #59 ? It might become important soon enough, corus disk consumption is increasing faster and faster.

michaelkaye commented 2 years ago

Object storage has the big downside that grep isn't efficient on it.

The search engine we have only indexes a few weeks at most because of the cost requirements to keep that going - if we remove the option to grep then we would need to replace with ... a lot of money for indexing servers.

benbz commented 2 years ago

OOI is grep a normal use-case? I'm mostly familiar with people using the links through from the issues

michaelkaye commented 2 years ago

It depends what we're doing with the rageshakes:

Given an issue looking at the problem is a common use, but the inverse is also used - given a problem, find other logs like it. The rageshake-searcher was a good start on that, but it doesn't really (economincally) scale to years of logs vs just "grep". Unfortunately it is a low-frequency operation, but if we want to and have moved to object storage it becomes impossible without additional indexing.

(that and it'd be a chunk of work to change it)