basho / riak

Riak is a decentralized datastore from Basho Technologies.
http://docs.basho.com
Apache License 2.0
3.93k stars 534 forks source link

Upgrade from 2.0.6 #985

Open andrewzeneski opened 5 years ago

andrewzeneski commented 5 years ago

Can Riak 2.0.6 be upgraded directly to 2.9.0 or are intermediate steps required?

martinsumner commented 5 years ago

Yes, there is a major customer going through 2.0.5 to 2.9.0 at the moment.

Note that the guidance here is still valid. If you intend to switch to Tictac AAE you may want to consider keeping the object_hash_version to legacy.

If you intend to change backend as part of the migration, you should first migrate to 2.9.0 on your existing backend (using the rolling upgrade approach), and then migrate backends using transfers.

andrewzeneski commented 5 years ago

Thanks for the quick reply Martin. Is there any documentation on backend migration? Specifically, eleveldb to leveled.

My cluster has been experiencing poor performance lately during times of heavy PUTs. It seems to resemble the "write amplification" issues documented in "Choosing a Backend". I am thinking of spinning up a test cluster and performing a full test upgrade/migration.

martinsumner commented 5 years ago

Essentially, assuming you're currently using intra-cluster AAE:

Step 1: Doing a rolling upgrade in the cluster to the new version, but maintain previous config settings (e.g. backend = eleveldb, anti-entropy active etc)

Step 2: Setup a new node with the new backend configuration, but with unchanged anti-entropy settings

Step 3: Do a cluster transfer to replace an existing node with the new node. This will transfer all data across. Do this node by node until all your nodes have been transferred

Step 4: Switch the AAE configuration to migrate to Tictac AAE (change riak.conf so that tictacaae_active is active but anti_entropy is passive and restart each node in turn.

We've found that transfers from eleveldb to leveled can be relatively slow, but clusters can be more stable with a higher transfer-limit. This is a key thing to test: how high a transfer-limit you can tolerate for Step 3.

@ramensen - do you have any further advice for Andrew?

martinsumner commented 4 years ago

@andrewzeneski is it OK to close this issue now?