Transitioning from Node Store to Shards

[miguelportilla]

From Node Store to Shards

Upcoming enhancements and new features will allow rippled to use shards instead of the node store as the primary database. Because shards offer many advantages over the node store, most operators eventually will prefer to use it as the sole database. But transitioning users from the node store to primarily using shards will likely span several releases and the transition will require carefully planned steps.

Configuring a server to use shards as the primary or sole database will be straightforward for servers that keep minimum history. But it is fair to assume that servers with a significant amount of stored history will not want to lose it. Stored node store history doesn’t have to be discarded. One approach to address the issue would be to automatically have rippled import the data into shards. The import would be time-consuming but occur as a non blocking background operation.

In the forthcoming shards, an enabled server will store the tip of the XRP ledger and continually build the current shard. When a current shard is completed, it replaces the previous shard and a new, current shard is commenced. At a minimum, all servers will store these two shards. Additionally, servers will also have the option of helping the network to maintain distributed XRP ledger history by specifying an amount of randomly selected and rotated shards to store. A possible feature to allow a server to hold a minimum number of previous shards may be needed.

The following proposed approach may be the first step in the transition. Enabling shards at this point would remain optional. The significant difference from today is that if enabled, shards would act as the primary database.

Proposed configuration possibilities.

• Node store only configured rippled operates as it does today.

• Shards only configured rippled operates using shards as the primary database.

• Shards and node store configured rippled operates using shards as the primary database. New acquired history will be stored only in shards, not the node store. The node store is read from as a fallback if requested data is not present in shards. The node store is automatically imported into shards and removed upon import completion.

I welcome your feedback on issues, solutions and further planning. Thanks!

[miguelportilla]

@MarkusTeufelberger this is an area you are very familiar with. I'd love to hear your thoughts on the matter. Thanks.

[mDuo13]

This makes sense to me.

[MarkusTeufelberger]

Personally I'd really love to first make sure that shards are fully deterministic, so instead of sending around small fetch_packs of historic data, servers can start sharing whole shard databases with confidence that they are complete and unmodified.

More recent history (the "current" and probably also the "previous" shard until its contents have been rewritten to a deterministic one) would still be served via fetch packs, deterministic shards though could be served by a ton of other means, even not using rippled at all.

I'm missing a way to decide which shard to throw away once limits are reached (it would really make more sense to say "I have 500 GiB of storage for historic data" instead of "store the last x ledgers") - either go the BitTorrent proven route of querying your peers and throwing away the most common ones first or just do the simple thing and delete at random.

In the end, there would be 2-3 types of node storage: "active" shards (data is appended as it comes in), "deterministic" shards (an active shard is sorted, re-written, index re-hashed and it is kinda the "archival" format of a shard) and the current node database (which is kinda like an active shard with 60 million ledgers...).

Since the active database/shard would not be huge anyways, it might even be possible to just keep it in RAM or in SQLite and only require a high performance database solution for the deterministic shards? Even then these are for the forseeable future going to be small enough to fit completely in RAM on larger servers, compared to the several Terabytes of a node_db currently, so there might not even be that much benefit in using a disk-only hash map such as NuDB.

[miguelportilla]

@MarkusTeufelberger thank you for your comments, I appreciate your input. There are several tasks being worked on that will need to be completed before the transition from node store to shard store can begin. One of those tasks is deterministic shards and another is the improved algorithm used to select what shard history to acquire (if any) along with the current and previous shards. These building blocks, along with a few others, need to be in place before the transition from node store to shard store commences. Assuming all of the features are in place and the shard store can be used to replace the node store, a transition plan will be needed to make things easy for users. The transition and the plan mainly affect servers that are storing history.

[MarkusTeufelberger]

Yeah, I hope that the amount of history that's available throughout the network will increase quite substantially with these changes. :-)

[carlhua]

@MarkusTeufelberger great feedback! Its always refreshing to get feedback from the community. I do have some questions that I hope you can provide input to:

1. Do you have any suggestions on how we transition a node store user to use shard store only? If tomorrow we have shards completed, what would be the transition look like? We have some ideas but would be great to hear from you.

2. Any suggestion on how we should release this to the network? Currently we are thinking perhaps we will start generating/making shards initially as phase 1 and then we will roll out the logic to replace nodestore with shardstore. We welcome any thoughts on this particular topic.

[MarkusTeufelberger]

My node store was migrated from hyperleveldb to rocksdb to nudb already, I guess migrating it to deterministic shards would be similar.

UX wise this will be a several day job for people with full history most likely, so having some indication of progress would be nice - OTOH there are only maybe a dozen of people and entities anyways out there who keep full history atm, so investing a lot of tooling into a one time one way migration doesn't seem too necessary. Maybe just watching manually that from time to time a new file pops up in the shard store is already enough.

I would expect to be able to take my current node store config, move it to the import_db section, configure the node_db section with shards and run rippled --import. There's already a --nodetoshard command as well, so maybe that could be used too (though I'd expect that the node_db section would NOT allow any more a non-sharded configuration, so this is likely more of an import job). For servers running the current default config, they would just need to change their config file. Importing there isn't really necessary, the online_delete settings that are in there by default are anyways far too tight to make it worth to help them migrating data. I would expect the server crashing and a warning to pop up similar to the one for https://xrpl.org/fix-sqlite-tx-db-page-size-issue.html at the most.

All in all I'd prefer the "rip the band-aid off" approach instead of a "we'll support both configs in the same file and beg users for a few releases to make the switch". Ideally this happens with a version that doesn't also contain changes that are expected to be quickly activated as amendment on the ledger (e.g. don't combine it with the release that contains Tickets and ideally activate Tickets asap/before this node store migration starts, so people can migrate on their own pace - alternatively delay Tickets by a few months). Since the code is already there anyways, you probably could support a --legacy mode (or even a --preview mode, so people can migrate already ahead of time, before the shard store becomes the default/enforced).

So my answer would be:

1) Is not a real pain point other than having to actually modify the default config file for existing installations with 100% default config (auto-update will be a pain there). At least deb packages already have a mechanism to deal with this, I suspect that rpm are similar. Replace the [node_db] section options with ones that make sense for a shard store and disallow existing settings. Default users don't serve much history and if they have any issues, they can just re-install and also be done with it. Add a setting with at least 50-100GiB historic shards to the default config, so the issue of too little history in the network changes over time too.

2) Roll out 1.7.0 with a --preview-flags=ShardStore option that already implements this change and make it the default in 1.8.0 adding a --legacy-flags=ShardStore option that restores the current behavior (general support for feature flags could come handy for other things too in the future, e.g. the GRPC stuff is currently also a bit like that). Keep support for this flag around until 1.10.0 and deprecate it there. For migrating large amounts of node database data, either use the --import (if the [node_db] section is exclusive to shard options) or --nodetoshard (if you really want to go through the mess of changing the [node_db] settings twice) functionality that already exists. There is maybe a dozen people out there that will need to use this and you'll likely already have one way or another to get in touch with them. After that (and even if some don't manage to do it) there should be a lot easier and efficient ways to get to full history from scratch compared to before (e.g. sharing shard stores via IPFS or BitTorrent - no more months long syncing!). Timeline would really depend on how fast you want Tickets out there and activated, I'd personally love to see that one up and running still in 2020, which would put the nodedb-->shardstore migration into Q1+2 2021.