search

rabbitmq / osiris

Log based streaming subsystem for RabbitMQ
Other
45 stars 10 forks source link

Optimise index scanning #144

Closed kjnilsson closed 1 year ago

kjnilsson commented 1 year ago

Optimise index scanning using a skip search approach. This improves worst case index scanning scenarios by making the scanning to find offset as well as epoch / chunk_id paris (as part of osiris_log:overview/1) up to 10x faster.

Move reader context caching earlier in writer initialisation

To increase the chance of a replica calling osiris_writer:overview/1 will get the cached reader context instead of having to call into the writer process.

Unfortunately we can only cache the context after log init but it can be done before any other work such as tracking recovery.

Make the replica call into the leader process (osiris_writer:overview/1) use infinity timeout to avoid cases where writer init takes so long a replica times out and has to be restarted.

osiris_log refactoring and log improvements.

kjnilsson commented 1 year ago

A real example of improvement from rabbitmq server logs:

BEFORE:
2023-10-03 10:06:19.533069+01:00 [debug] <0.26360.0> osiris_log:offset_idx_scan/2 completed in 0.032147s

AFTER:
2023-10-03 10:28:52.038459+01:00 [debug] <0.1252.0> osiris_log:offset_idx_scan/2 completed in 0.001742s