OS Optimization settings

[danieldreier]

I'd like to implement optional OS optimization settings, similar to what @Cornellio did in https://github.com/Cornellio/puppet-riak/blob/master/manifests/baseconfig.pp. I'm still hoping to support non-linux platforms, so a lot of this will need to be wrapped in a case statement.

I haven't found a great single source of recommendations. As source material, what I'm aware of it:

• http://docs.basho.com/riak/1.3.2/cookbooks/Linux-Performance-Tuning/#Linux-Tuning
• http://docs.basho.com/riak/1.3.2/cookbooks/Performance-Tuning-AWS/
• http://docs.basho.com/riak/1.3.2/cookbooks/Open-Files-Limit/
• http://docs.basho.com/riak/latest/ops/tuning/linux/

I'm not clear on how much the riak 1.3 recommendations still apply to riak 2. I'm assuming that IO patterns should be relatively similar, but I don't really have a lot of basis for that assumption.

My current POC looks like https://gist.github.com/danieldreier/a9a20e2dee7d2db955c2 (heavily based on @Cornellio's work) but that doesn't even really work yet, more of a scratch pad for ideas.

Are there other must-read sources? I didn't see anything in the little riak book about it.

[Cornellio]

I'm working on ulimits and sysctl configs and hope to have something to show you soon.

The docs on tuning are a little vague in areas, yet the version 2 guide still has similar guidance:

• http://docs.basho.com/riak/2.0.5/ops/tuning/linux/

Great work on getting this into the forge. It looks great!

[danieldreier]

@Cornellio - awesome! A few more resources:

• https://github.com/basho-labs/riak-chef-cookbook/blob/master/attributes/default.rb is the stuff being managed in the chef cookbook.
• @CompareGroup was awesome enough to share their own puppet code they use to manage Riak at https://github.com/CompareGroup/cg_riak/blob/master/manifests/init.pp - it contains more similar settings.

One challenge I'm not sure how to solve, off hand, is that different people will use different mount points, so we can't just manage filesystem settings on /. The best I can come up with would be to manage the Riak file locations as a parameter / set of parameters, then add a custom fact to the module (or create a new module and require it) that lists mounted filesystems as facter facts. We can then iterate over the mounts and (potentially) optimize settings on the mounted volumes that contain Riak data and are of the correct filesystem type.

[Cornellio]

I like the idea of using facts to locate where riak data is mounted and only apply filesystem setting there.

I notice that ring_size is not yet configurable, which would mean getting a default of 64. Setting the ring_size at time of cluster creation is important because it cannot be changed afterwards without losing data. It has to be chosen carefully due to scaling and performance.

Reference:

http://docs.basho.com/riak/latest/ops/building/configuration/#Ring-Size

With the riak admin features, It might be enough to just enable Puppet to join members into a cluster. If cluster operations need to happen after you have data or in response to problems, it probably would be complex enough that you have to do it by hand anyway. I've run Riak for over a year in production and it's mostly been a black box. We had one issue due to faulty hardware where we need to replace a node. I ran into some pain reinstalling the host and finally could rejoin the cluster by hand. Another issue is the setting of an expiry on data. It's set for an entire bucket at the time of installation and can't be changed afterward without creating a new bucket and losing data, so we have no way to purge and free up disk space. Beyond that, it's been very reliable and fast.

[danieldreier]

That's really useful feedback about the problem scenarios you've run into.

ring_size can be set at cluster creation time with (for example):

class { '::riak':
  settings       => {
    'ring_size' => '128',
  },
}

I believe that Basho's basic configuration docs are out of date, because Riak 2 supports ring resizing under load unless you're using Riak Search 2.0, strong consistency, or multi-datacenter replication.

I'm pretty convinced that we can automate common cluster management operations; it should even be possible to specify in puppet DSL that a node is a replacement for another node and handle that in a safe, predictable way. It'll require implementing a bit of conditional logic, checks, etc.