Read persistent data in reverse order

[amclain]

When using disk persistence, after restarting the database and refreshing the UI before the logs had finished being read (it takes a couple minutes), I noticed the graph displayed the oldest values but was missing the most recent:

Of course, this makes sense logically because the log is read from oldest to newest. However, from a UX standpoint (or at least most that I can think of) the recent value is the most relevant and the older values get less relevant the further back in time you go. In the graph above the measurements for the day are not present, but data from a few days ago is. I think the same holds true for other data types like MVREG: The last known value is expected when reading the logs. Although by the nature of a database that favors availability it's acceptable for values to bounce around when the log is being read, in practice my concern is that it may be unexpected and the application may appear "buggy".

I'm not sure how difficult this would be to implement, so at this point I just wanted to point it out and see what you think. I don't think it's an urgent priority. I think the issue could also be avoided for now by having a cluster and performing a rolling restart so that the latest values are synced from the other nodes.

[jemc]

So, for context, persistence is currently done with append-only log files, where all operations that mutate state get their delta-state CRDT added to a log for that keyspace. This is very similar in concept to how Redis AOF files work (see Redis docs on persistence for more info). So, in other words, the data becomes available in the same it became available when first being written, following each of the same operations that were observed. Note that this may be different in practice from the order of timestamps in your TLOG instance.

The append-only format would make it difficult to write the data in reverse order (it would become prepend-only at that point, which wouldn't have the same desirable characteristics in terms of dealing with file handles). It would also be fairly costly to try to read the data in reverse order, and doing so would significantly increase the time it takes to load the data from disk in general.

I think there are two general ways of alleviating this pain point, which would be better options:

• :one: Adding some kind of new SYSTEM command that lets you check when the disk data is fully loaded.

  • This could then be observed by the application when managing the pool of cluster connections, to avoid sending other commands to an uninitialized node.
  • Doing this kind of model would also assist with other potential issues of directing production traffic to a node that is still initializing, like the fact that such traffic might have longer latencies while competing with the disk restore work.

• :two: Periodic compaction of the append-only log files, which would have a few desirable effects:

  • Data after compaction would take up an amount of space on disk that is proportional to the amount of data in the database, instead of taking up a grow-only amount of space on disk with lots of additional overhead from storing the individual (sometimes overlapping) deltas.
  • The overall time to load the data back into the database would be significantly faster.
  • Data from a single large TLOG instance would be loaded all-at-once instead of incrementally, avoiding the original problem you talk about with not all data being available yet.

[amclain]

Both of those options sound great. It also doesn't sound like they're mutually exclusive and at some point both features could exist.