search

onemoredata / bagger

Massive log storage in PostgreSQL
BSD 2-Clause "Simplified" License
9 stars 0 forks source link

Storage agent part 2 #105

Closed einhverfr closed 3 months ago

einhverfr commented 7 months ago

Fixes #17 Fixes #31 Fixes #95

This is the second piece of the storage agent. It involves some refactoring to encapsulate some important logic in the Schaufel process, and provides full Schaufel process control as well as data retention enforcement.

This agent constitutes a general event loop watching for Schaufel's exit, as well as arrival of new data from the kvstore system.

This is intended to be reviewed after #102, but the two could be reviewed together if people find that easier. It is not marked as ready for review yet because tests have not been added but review of form, idea, etc, is welcome at this stage.

einhverfr commented 7 months ago

After significant thought I have decided integration tests are not possible at this time because these require more infrastructure than we have. However we have tests for all of the pieces. I therefore think it is ready for review.

einhverfr commented 7 months ago

These cannot be merged yet because of issues with the lw_integration tests caused by changes here. The answer is to fix the lw_integration tests, but I think the fixes are more general than in this PR, so let;s defer this one for now.