IDEA: Refactor how we do puppet modules

[Atchuu]

See: http://projects.puppetlabs.com/projects/1/wiki/Puppet_Best_Practice2

I was thinking, it would probably be much easier to create new flavors if we treat each entity as its own module...

For instance, currently we install this httpd and apache and whatever else in the manifests.pp file, while this is entirely valid, wouldn't it be more friendly to future flavors (the original idea behind minus-r and minus-d) if we broke each separate entity out into it's own module, and load them on a I want this but not that in the Vagrantfile.

If this is more ideal, I also planned on writing a bash/batch script to help users select what they want, along with that would be a fresh RHEL 5 VirtualBox VM. I have the ISO at home, it does not contain any extra things on it, it is merely a clean, non-RHN'ed iso. So we could use it as a real base for our testing.

[jlank]

+1. Also before you write a whole bash script to do an interactive build/configuration step, explore to see if there are any existing build tools we could use like grunt or rake or something along those lines so it can benefit from a community and documentation.

[jlank]

Also, we'd be smart to check out puppet forge and see what modules we can pick up from there, and contribute back to http://forge.puppetlabs.com/

[gregelin]

I like the idea of regular refactoring. I also like avoiding pre-mature optimization. A little refactoring to pull apart tomcat from apache, etc. seems to make sense. Creating a script for users selecting configuration could be premature. I find it a mentally easier to choose among pre-configured, well-named vagrant boxes than to to have a universal vm which I then choose features. I think there is advantage in being able to tell distributed colleagues to "just get box minus-r-federal" and know what they will get. Otherwise there's a handoff: "get box minus-r-federal and then select ..."

All of that said, the cool thing about this approach is we can form, branch, and try different things! I say let's keep the main branch simple and robust and either use forms or branches for some experimentation.

[jlank]

Agree

Sent from my iPhone

On Feb 25, 2013, at 6:40 PM, Greg Elin notifications@github.com wrote:

I like the idea of regular refactoring. I also like avoiding pre-mature optimization. A little refactoring to pull apart tomcat from apache, etc. seems to make sense. Creating a script for users selecting configuration could be premature. I find it a mentally easier to choose among pre-configured, well-named vagrant boxes than to to have a universal vm which I then choose features. I think there is advantage in being able to tell distributed colleagues to "just get box minus-r-federal" and know what they will get. Otherwise there's a handoff: "get box minus-r-federal and then select ..."

All of that said, the cool thing about this approach is we can form, branch, and try different things! I say let's keep the main branch simple and robust and either use forms or branches for some experimentation.

— Reply to this email directly or view it on GitHub.

[gregelin]

@atchuu I'm beginning to reconsider that a shell script is pre-mature optimization. I just ran into an issue where I tried to vagrant up a second instance of minus. Second instance failed because the ports were being used by the first instance. Made me think a check and an option to select a different ports would be useful.

[nihonjinrxs]

I agree with all this approach. It's definitely a good idea to refactor regularly and in small bits. If we do this "the Ruby way," you could do the scripting (however you do it) to allow for parameters ("I want apache to run on port X and forward to port Y") but have smart defaults (checking for the port to be used and incrementing until you find an open one).

As to breaking the puppet manifests into small bits, I like the idea, particularly if we set it up so that I can have a single main manifest for configuration and just comment things out as needed if I decide I want something different.

The Ruby/Rails rule of Convention over Configuration is exactly this: use convention and have smart defaults where there's no need for the user to set something, but allow for configuration where they want to do something out of the ordinary.

[Atchuu]

Was thinking more about this and I like the idea of using node.js to script this stuff, that way it is cross platform and guaranteed to be installed on the systems that people are using, allowing us to create and maintain scripts once without any extra tools added to the stack.

[gregelin]

Would Nodejs have to be on every machine or just on our tools that create the GMa?

Why Node vs Go which seems to be getting lots of attention? Or Ruby which is the heart of puppet and Chef?

Lastly, what would be a couple simple use cases to try out?

Greg

Sent from my iPhone

On May 27, 2013, at 9:39 PM, Blaine notifications@github.com wrote:

Was thinking more about this and I like the idea of using node.js to script this stuff, that way it is cross platform and guaranteed to be installed on the systems that people are using, allowing us to create and maintain scripts once without any extra tools added to the stack.

— Reply to this email directly or view it on GitHub.

[jlank]

I would be open to using(learning) Go. The trade offs that come to my mind for node - pure js modules (excluding modules that have c/c++ bindings) run on the vm, so we need to have this installed on every machine. Go is compiled, so if we do cross platform we may need to pre-compile for our targets to speed builds/deploys up. The benefit of using Ruby is we have it for free, also cuts down on the install/bootstrap of each machine. Drawback to that is I think it is considerably slower than go/node.

[Atchuu]

I expected the scripts would live on the same box as the vagrant machine, allowing you to create dynamic vagrant/manifests via options (that could be shared) and no other scripts would be needed. In this use case node is already installed (vagrant dependency) and users would not have to install any other tools to use our stuff. If we used something else we would then require more dependencies for gitmachines just for our scripts.

John Lancaster notifications@github.com wrote:

I would be open to using(learning) Go. The trade offs that come to my mind for node - pure js modules (excluding modules that have c/c++ bindings) run on the vm, so we need to have this installed on every machine. Go is compiled, so if we do cross platform we may need to pre-compile for our targets to speed builds/deploys up. The benefit of using Ruby is we have it for free, also cuts down on the install/bootstrap of each machine. Drawback to that is I think it is considerably slower than go/node.

---

Reply to this email directly or view it on GitHub: https://github.com/OpenDataDevOps/minus/issues/11#issuecomment-18566552

[nihonjinrxs]

Go is probably faster, being a compiled language. I'd be extremely surprised if node was any faster than Ruby, and you don't get it for free on any OS. In fact, I know of no OS that comes with node and npm installed, whereas only Windows doesn't have Ruby. JavaScript in the browser is not the same as node. As to Go, I think we should beware of flavor-of-the-day languages that very few people use. We want these machines to serve as an easy starting point for folks. Having to learn a new programming language doesn't qualify as easy in my book, even for talented developers.

[Atchuu]

I guess I'm confused as to why we are discussing alternatives.... If Node will do everything we need scripting wise, and it's installed because of Vagrant, then why worry about making sure anything else is installed?

I am weary about applications that have too many dependencies.... because there ismore potential for versuib conflicts... I don't expect speed to be a problem either as I can't see this being anything but a manual process.

Using a script, I should be able to build a custom vagrantfile and manifest (or whatever else we use for building if not puppet) but all of the choices (which could be hundreds) would require the user to make decisions on what they want in their VM....

Think al a carte VM building...

USE CASE: I would like tomcat, apache, postgresql, node, java, openssl, I should be able to execute a script that will prompt me for which of the offered resources I want to include, and any pre-configuration I may want, the results generating a Vagrantfile and a manifest that I can then execute using Vagrant.

This is not something that needs to be installed on the VMs as I see it as something for creating the vagrantfile and manifests before you vagrant up the vms.

Does that make sense?

[jlank]

The Node/v8 vm is faster than the ruby. I can look up some benchmarks for the group for comparison purposes (though the benchmarks may not be applicable to our use case - that being said we should hypothesize what our bottleneck will be). It depends on what context we are applying 'fast' to. Do we need fast I/O, fast CPU processing, or something else?

Sent from my iPhone

On May 28, 2013, at 9:41 PM, "Ryan B. Harvey" notifications@github.com wrote:

Go is probably faster, being a compiled language. I'd be extremely surprised if node was any faster than Ruby, and you don't get it for free on any OS. In fact, I know of no OS that comes with node and npm installed, whereas only Windows doesn't have Ruby. JavaScript in the browser is not the same as node. As to Go, I think we should beware of flavor-of-the-day languages that very few people use. We want these machines to serve as an easy starting point for folks. Having to learn a new programming language doesn't qualify as easy in my book, even for talented developers.

— Reply to this email directly or view it on GitHub.

[jlank]

I'm a little confused. Is node a dep of vagrant? Also npm is a solid package manager for dep resolution, so I'd put that in he pro column

Sent from my iPhone

On May 28, 2013, at 9:51 PM, Blaine notifications@github.com wrote:

I guess I'm confused as to why we are discussing alternatives.... If Node will do everything we need scripting wise, and it's installed because of Vagrant, then why worry about making sure anything else is installed?

I am weary about applications that have too many dependencies.... because there ismore potential for versuib conflicts... I don't expect speed to be a problem either as I can't see this being anything but a manual process.

Using a script, I should be able to build a custom vagrantfile and manifest (or whatever else we use for building if not puppet) but all of the choices (which could be hundreds) would require the user to make decisions on what they want in their VM....

Think al a carte VM building...

USE CASE: I would like tomcat, apache, postgresql, node, java, openssl, I should be able to execute a script that will prompt me for which of the offered resources I want to include, and any pre-configuration I may want, the results generating a Vagrantfile and a manifest that I can then execute using Vagrant.

This is not something that needs to be installed on the VMs as I see it as something for creating the vagrantfile and manifests before you vagrant up the vms.

Does that make sense?

— Reply to this email directly or view it on GitHub.

[nihonjinrxs]

So I've seen benchmarks of Rails vs Node.js for web apps, but that's a bit of a different story. Rails is a pretty heavy framework. In any case, all the puppet and chef stuff is built in Ruby, so we should have it anyway, right? I'm not opposed to Node.js. I'm just wary of adding more dependencies where they aren't really needed.

[Atchuu]

I thought node was a dep of Vagrant, maybe I was thinking of ruby, in which case, I think we should script in ruby.

I am very against the idea of requiring people to install another tool just to build custom VM setups... very.

[nihonjinrxs]

Agreed. I think you were thinking of Ruby, and if we're talking about compiling scripts from script fragments, there's no bottleneck there to worry about. Lots of folks use Ruby for sys admin style scripting with no problems.

[jlank]

I agree with you there. I didn't think node was a dep of vagrant- if it is we could use either or, at that point it's a preference choice.

Sent from my iPhone

On May 28, 2013, at 9:55 PM, "Ryan B. Harvey" notifications@github.com wrote:

So I've seen benchmarks of Rails vs Node.js for web apps, but that's a bit of a different story. Rails is a pretty heavy framework. In any case, all the puppet and chef stuff I'd built in Ruby, so we should have it anyway, right? I'm not opposed to Node.js. I'm just wary of adding more dependencies where they aren't really needed.

— Reply to this email directly or view it on GitHub.

[nihonjinrxs]

Vagrant is built 99.9% in Ruby, according to GitHub: https://github.com/mitchellh/vagrant

[Atchuu]

Okay, I confused people, I apologize. This idea was merely for creating custom configurations from, as you said, fragments... Ideally we would create new pre-configured VMs using this method, to yet again "eat our own dog food" but treating the tools we install as objects and each configuration block as methods of that object, much in the way object oriented programs allow for inheritance and the like.Doing it like this means we should probably design how the fragments should be set up to ensure maximum support (things like optional blocks say for apache port configurations) and manditory blocks (such as constructors) which ensure the apache tool is installed. I think treating them like this is the best way to make it very powerful without any duplication of effort... Agree or disagree?

Meaning if we updated the apache "object" all I would have to do is update, and rebuild my vagrantfile/manifest, which can be very very powerful.

Compiling might be the wrong word, maybe aggregating?

[nihonjinrxs]

I agree. That's what I had in mind too. How about coalescing?

[Atchuu]

That's fine :)