Use memory buffer to write current deltas before flushing them

mkabilov / logical_backup

[WIP] Continuous backup of the tables using PostgreSQL logical replication

MIT License

9 stars 3 forks source link

Use memory buffer to write current deltas before flushing them #11

Closed alexeyklyukin closed 6 years ago

alexeyklyukin commented 6 years ago

Right now we write each delta to a delta file right away. What we could do to improve the performance is to write those deltas to the memory buffer first, flushing them when we fill the file.

As a related note, we could also send the flush response back to the server once we write all deltas to the file, fsync and close it. By not doing fsync we risk losing some records on crash, since the server usually disposes of them after receiving the flush.

mkabilov commented 6 years ago

two points here:

flush delta memory buffer to the file as soon as we get sigterm(and maybe sigusr1?) signal
maybe it will make sense to get rid of the tempDir, and flush data directly to the archiveDir. Also there should be monitoring(most likely prometheus http endpoint) to see the number of the buffer files which need to be flushed to the archiveDir

alexeyklyukin commented 6 years ago

Fixed at https://github.com/ikitiki/logical_backup/pull/16