search

basho-labs / puppet-riak

A puppet module to deploy Riak clusters
Apache License 2.0
33 stars 37 forks source link

Support Riak 2.0 #26

Closed haf closed 9 years ago

haf commented 10 years ago

/cc @jaredmorrow

New config - need specification.

Suggest that we stop supporting puppet 2.7 and its hiera in this next version.

perbergland commented 9 years ago

Still nothing one year later?

dmitrizagidulin commented 9 years ago

@perbergland-meltwater - We generally don't use Puppet here, and so simply haven't had internal Puppet expertise available to update. (This reminds me - since this is an unofficial community-supported repo, we should move it to basho-labs instead of basho).

However, take a look at @mchesler's commits (on Mar 25, 2014 https://github.com/mchesler/puppet-riak/compare/basho:master...master ) or @fpgeek's riak2 branch https://github.com/fpgeek/puppet-riak/compare/riak2 , to see how they updated it to work with Riak 2.0

haf commented 9 years ago

They could send a PR?

mbbroberg commented 9 years ago

@danieldreier & @haf as mentioned on #48, this is a major update and - to me - the main goal. Let's use this (overdue) conversation to keep track of progress on 2.x

mbbroberg commented 9 years ago

Requires feedback from #51 before it can progress to Ready state.

danieldreier commented 9 years ago

I put together an experimental riak 2-only module last night to see what could be done if we only support puppet 3.7 but without a future parser dependency. It installs riak 2 on CentOS 6 and Debian Wheezy and has pretty extensive rspec-puppet test coverage and basic beaker test coverage. I don't intend that as a fork of this project, but part of me wonders whether it might make sense to build a new riak 2 module in parallel to maintaining this one.

I'm hesitant to make sweeping PRs that make breaking changes to this one, because I'm assuming that 100% of the users of this module use it for Riak 1, and nearly 100% of the new users who are looking for a riak module will be looking to use Riak 2.

mbbroberg commented 9 years ago

@danieldreier I'd be glad to cut a version as it stands right now with 1.3.x support and then move forward with the 2.x goal. That's absolutely the intention from a PM POV.

Cornellio commented 9 years ago

I agree with moving forward with a 2.x only module, but a little concerned about only supporting puppet 3.7+. We are still using 2.7, although we are planning to upgrade soon.

So If I were to contribute, is this danieldreier/riak2 module the one to use now?

I'm a bit new to the Github workflow. Would these be the basic steps to follow?

Is that correct?

danieldreier commented 9 years ago

@Cornellio I'm really only using my danieldreier/riak2 module as a development testbed, to see what I can do without being constrained by riak 1 or older puppet version compatibility, and without having to add tests onto older code; I'm hoping to wholesale forklift that code into this one after meeting feature-parity and incorporating the features that your and other modules have that I lack. For example, I've been realizing that the system settings are less optional than I had thought.

I should really add a CONTRIBUTING.md file documenting this. The workflow I'd suggest is:

Regarding puppet 3.7, puppet 4 is going to be here sooner than people think. I don't have a specific timeline but I think we're looking in the several-months-out timeframe, because there are very usable pre-release versions and the updated language is (basically) feature-complete in puppet 3.7 if you enable future parser. If you're coming from puppet 2.x, you might considering just jumping straight from your current codebase to getting compatible with 3.7 with future parser enabled, because in 4.x that'll be the only option.

Does that make sense?

Cornellio commented 9 years ago

I think that makes sense. For the rspec and beaker tests, they ca be run in vanilla Debian and CentoOS Vagrant boxes?

On Mar 26, 2015, at 10:04 AM, Daniel Dreier notifications@github.com wrote:

@Cornellio I'm really only using my danieldreier/riak2 module as a development testbed, to see what I can do without being constrained by riak 1 or older puppet version compatibility, and without having to add tests onto older code; I'm hoping to wholesale forklift that code into this one after meeting feature-parity and incorporating the features that your and other modules have that I lack. For example, I've been realizing that the system settings are less optional than I had thought. I should really add a CONTRIBUTING.md file documenting this. The workflow I'd suggest is: Fork repository Create feature branch Hack Run rspec-puppet tests locally (rake spec) Run beaker tests locally (BEAKER_set=debian-76-x64 BEAKER_destroy=onpass BEAKER_provision=yes rake beaker) Commit Push Submit a PR (discussion can happen on the PR, but if there is a related issue link to it from the PR) travis-ci will run rspec-puppet tests against a wider range of puppet versions and settings and will post the result on your PR. Regarding puppet 3.7, puppet 4 is going to be here sooner than people think. I don't have a specific timeline but I think we're looking in the several-months-out timeframe, because there are very usable pre-release versions and the updated language is (basically) feature-complete in puppet 3.7 if you enable future parser. If you're coming from puppet 2.x, you might considering just jumping straight from your current codebase to getting compatible with 3.7 with future parser enabled, because in 4.x that'll be the only option. Does that make sense? — Reply to this email directly or view it on GitHub.

danieldreier commented 9 years ago

rspec-puppet and beaker tests are actually run on your host machine. Beaker creates its' own vagrant boxes to run tests inside of, and rspec-puppet uses the puppet you have installed on your development environment to build catalogs. rspec-puppet basically compiles the puppet code into a catalog and then inspects the catalog for certain conditions, whereas beaker actually tries to set up a virtual machine and then use serverspec to verify that the VM is running the expected services, has the right ports listening, etc.

One other step I forgot to add was to add test coverage for any new changes. If it's a straight up refactor of existing code, no new tests are needed. If there's new conditional logic being added or new features, the tests should fail without the code changes and pass with. Conditional logic is best tested in rspec-puppet and full-on integration testing is best done in beaker. If you need assistance in figuring out how to get that test coverage I'd be glad to help.

Cornellio commented 9 years ago

Thank you, that makes sense and gives me a good opportunity to adopt a proper testing framework.

Cornellio commented 9 years ago

@danieldreier Riak2 initial version looks good.

In trying to setup my environment for beaker-rspec, I already have Vagrant with the vmware fusion provider, but I see most module tests are geared toward virtual box. I'm trying to get the fusion provider working but have no luck so far following this guidance. Not sure if it's worth the effort as virtual box is so much more common in terms of collaboration.

danieldreier commented 9 years ago

The beaker docs are pretty spotty; I'm sorry you're having a hard time with it. I've never used it with vmware fusion, but my vagrant - vmware fusion experiences have been poor enough that I switched back to virtualbox.

If you're using either my riak2 module or the PR I just made to merge that into this repo, you should be able to:

bundle install
BEAKER_set=centos-66-x64 BEAKER_destroy=onpass BEAKER_provision=yes bundle exec rake beaker

You will probably want to disable the virtualbox vagrant plugin first; I've had nothing but trouble with it. I'd also recommend you use the vagrant-vbguest and vagrant-cachier vagrant plugins.

The beaker tests will be slow (10 minutes?) with virtualbox or vmware because it has to provision and destroy a VM for each test run. I'd suggest running it once at the beginning of the development session to validate that tests pass when you start, and another run right before you make a PR.

When I get it set up myself, I'll add docs to the CONTRIBUTING.md on how to get it set up with GCE or AWS, since that may be a little faster.

mbbroberg commented 9 years ago

Just FYI to this thread that a PR is up for review: https://github.com/basho-labs/puppet-riak/pull/52

Cornellio commented 9 years ago

@danieldreier Thank you, I switched to virtualbox instead of fusion and beaker runs now. It's an impressive piece of automation.

On your riak2 module the test process runs completely using bundle exec rake beaker, although I am seeing failures in the module itself (both on CentOS and Ubuntu.) I'll explore more for issues on my end.

I'm also able to run a test cycle with beaker-rspec (against puppetlabs-apache module) with bundle exec rspec spec/acceptance

danieldreier commented 9 years ago

I'm not sure the issue is necessarily on your end. I encountered issues with it as well. For context,

I've been doing most of my work using the internal puppetlabs VM pooler, which is a godsend -- it keeps a constantly-running pool of about 1000 VMs running that beaker pulls out of the pool, provisions, tests and sends back, which makes beaker tests take about 45 seconds.

For some reason it's been passing on that platform but not in virtualbox. Just a guess but I suspect that there's a timing issue - the pooler hardware is incredibly fast. My hunch is that riak starts a lot faster on the pooler hardware than it does on a local virtualbox system, and in virtualbox it's not ready for tests to run by the time they start, even though I tried to build in a delay for that.

danieldreier commented 9 years ago

@Cornellio I think I figured out what was breaking tests in virtualbox and fixed it - look at commit https://github.com/danieldreier/puppet-riak/commit/045a240c356adbf2ee1b49f3ca384d53bdff38d8 in https://github.com/basho-labs/puppet-riak/pull/52

Cornellio commented 9 years ago

Yes indeed, it works fine now.

I'm thinking of adding system tuning settings with the kernel params and ulimits. It'll take me some time to get accustomed to rspec though...

danieldreier commented 9 years ago

https://github.com/basho-labs/puppet-riak/pull/52 was merged, so this is resolved. The module now only supports raik 2.