The future of knife-solo

[tmatilai]

One big step towards deprecating chef-solo was taken yesterday when chef-client --local-mode made it into Chef's master branch. The option starts a local chef-zero server which reads cookbooks, roles, nodes, data bags, etc. from the file system hierarchy ("kitchen"). It also supports search, and saving the node back to disk.

Sounds cool, doesn't it? =)

But this opens the question what will happen to knife-solo. (Another opening about the future was in #246, but in my opinion there is still need for provisioning servers without Vagrant :))

My earlier idea was to run chef-zero on the workstation, and then configure chef-client on the node to use it through an SSH tunnel. But synching the kitchen and running chef-client --local-mode sounds much simpler.

There are still some tasks that are needed:

1. Installing Chef to the node
2. Running the cookbook manager locally (Berkshelf, Librarian-Chef)
3. Uploading the kitchen with rsync (or pulling it from e.g. git)
4. Running chef-client on the node
5. Updating changes to the local kitchen (or to git)

I don't even see much need for Knife any more. A better fit could be integration with the next gen bootstrapper.

I'm interested to hear what @matschaffer and everyone else are thinking. Pinging also some involved Opscode staff if you already have some plans: @jkeiser, @danielsdeleo, @lamont-granquist

[tmatilai]

http://jtimberman.housepub.org/blog/2013/10/10/preview-chef-client-local-mode/

[matschaffer]

Syncing the kitchen and running chef-solo --local-mode sounds fantastic. It's actually the first way I envisioned knife-solo working but thought that re-implementing the chef api would be too much work to take on.

I haven't used bootstrapper but so long as it can work in a chef-solo mode I think that'd be viable. My main reason behind knife solo prepare was that knife bootstrap required advance knowledge of the system and required writing a template for chef-solo compatibility.

If chef-solo local mode and bootstrapper turn out to be viable replacements for solo-cook and solo-prepare, I'd be happy turning knife-solo into a wrapper around the two. The best code is no code at all ;)

[danielsdeleo]

Hi there. Bootstrapper isn't on the front burner at the moment, but it may become more feature complete as we start to use it internally for a handful of use cases that don't work well with the bootstrap template case. Eventually, I would like it to support a provision step that happens before the actual bootstrap, and then all the knife cloud stuff can become just a small plugin for bootstrapper.

So, I don't know what our timeframe looks like, but I think chef-solo support is absolutely a reasonable thing to have in that project.

[tmatilai]

Hi Dan,

thanks a lot for taking the time to comment here! I already skimmed a bit through the current code and when I find a bit of time and inspiration I'll try to spike something with chef-zero. I really like the concept as it would reduce a lot of the configuration and synching code the we now have.

[jayofdoom]

I honestly would like there to remain chef-solo support too. chef-zero can do writeback, which seems undesirable with knife-solo use cases.

[tmatilai]

@jayofdoom my wild guess (without any inside information) is that Chef 13 or so won't ship with chef-solo any more. =)

As a matter of fact, maybe half a year ago I hacked a proof-of-concept writeback in knife-solo by monkey patching node.save support to chef-solo and fetching the node data back to the kitchen. For sure I still have the code somewhere. I can really see a use case for that too, combined with chef-solo-search.

On the other hand, even if chef-zero supports writeback, there is no need to pull the changes to the kitchen unless wanted. So it can still be optional (with whatever the default is).

And don't get me wrong. I'm not trying to kill knife-solo or making it to stop support older Chef versions. Just planning for the future. I just hope to find the time to play around a bit more with chef-zero and bootstrapper.

[jkeiser]

It will not be hard to add an option to turn off write back in chef zero. Ice heard a few concerns of this nature and it seems like a good idea. On Nov 7, 2013 5:23 PM, "Teemu Matilainen" notifications@github.com wrote:

@jayofdoom https://github.com/jayofdoom my wild guess (without any inside information) is that Chef 13 or so won't ship with chef-solo any more. =)

As a matter of fact, maybe half a year ago I hacked a proof-of-concept writeback in knife-solo by monkey patching node.save support to chef-solo and fetching the node data back to the kitchen. For sure I still have the code somewhere. I can really see a use case for that too, combined with chef-solo-search.

On the other hand, even if chef-zero supports writeback, there is no need to pull the changes to the kitchen unless wanted. So it can still be optional (with whatever the default is).

And don't get me wrong. I'm not trying to kill knife-solo or making it to stop support older Chef versions. Just planning for the future. I just hope to find the time to play around a bit more with chef-zero and bootstrapper.

— Reply to this email directly or view it on GitHubhttps://github.com/matschaffer/knife-solo/issues/297#issuecomment-28023572 .

[runningman84]

I really like the idea of having chef zero on the workstation and using chef client in local mode through the tunnel.

Getting rid of rsync and having a writeback would be a gread benefit. Right now all my nodes, cookbooks and databags get synced. But only a small amount of these objects are really needed on each target.

[matschaffer]

Near idea! Would definitely make write backs simpler if they happened right on the workstation.

On Friday, December 13, 2013, runningman84 wrote:

I really like the idea of having chef zero on the workstation and using chef client in local mode through the tunnel.

Getting rid of rsync and having a writeback would be a gread benefit. Right now all my nodes, cookbooks and databags get synced. But only a small amount of these objects are really needed on each target.

— Reply to this email directly or view it on GitHubhttps://github.com/matschaffer/knife-solo/issues/297#issuecomment-30523178 .

-Mat

about.me/matschaffer

[jkeiser]

I actually deployed something that started up chef-zero on the host, and then tunneled the chef server port back to it through ssh. It works pretty well. net-ssh supports it natively. https://github.com/jkeiser/chef-metal/blob/master/lib/chef_metal/convergence_strategy/precreate_chef_objects.rb#L26-L28 https://github.com/jkeiser/chef-metal/blob/master/lib/chef_metal/transport/ssh.rb#L80

[matschaffer]

Nice! What are your goals with knife-metal? Sounds like it's doing some things similar to knife-solo.

On Friday, December 13, 2013, John Keiser wrote:

I actually deployed the tunneling idea on chef-metal, and it works pretty well. net-ssh supports it natively. https://github.com/jkeiser/chef-metal/blob/master/lib/chef_metal/convergence_strategy/precreate_chef_objects.rb#L26-L28 https://github.com/jkeiser/chef-metal/blob/master/lib/chef_metal/transport/ssh.rb#L80

— Reply to this email directly or view it on GitHubhttps://github.com/matschaffer/knife-solo/issues/297#issuecomment-30555984 .

-Mat

about.me/matschaffer

[jkeiser]

I'm on my phone, so instead of typing it out I will link to it: https://github.com/jkeiser/chef-metal

[makern]

I can see a whole bunch of security issues with allowing a server to tunnel back to my workstation. In fact the chef-zero readme states in the first paragraph that

It is NOT intended to be secure [...]. It does NO input validation, authentication or authorization.

Until major steps have been taken on that front I don't see why anyone would want to run it locally. Running it on the server after rsyncing the files seems fine though.

[lamont-granquist]

attacker on the server could bind to the high numbered port that it uses and use that to escalate privs. that might already be mitigated, though, if the client process waits for the server to spin up and will correctly exit if the server fails to bind. and this is a quite conceivable attack vector: consider a shared server with college students having user accounts and wanting to manage the server with chef-zero.

[makern]

Good point. I guess this could be prevented by having a shared secret between chef-zero and the client which can be used for authentication. Or, in fact, use proper SSL for the connection.

[jkeiser]

The process will not start unless chef-zero has already started and bound to the specified port. Otherwise an exception is thrown when chef-zero starts.

[pwelch]

Has there been anymore thoughts on the future of knife-solo and chef-zero?

I think knife-solo solves a use case and it would be nice to keep the project moving forward.

[matschaffer]

So after hearing the recent foodfight show on chef metal, I can't help but wonder if that will replace knife solo. Have yet to try it myself, but supposedly it uses a local chef zero accessed from the target node over an ssh tunnel. Sounds like worth a look for ideas if not adoption.

[lamont-granquist]

Yeah, the use cases of knife-solo should be pushed into chef-metal / chef-client -z / chefdk. The chef-client team definitely has zero interest or time to put in workflow tools around chef-solo and view that whole idea as a dead end and will aggressively push chef-solo users to stop using it and switch as a solution. The chef-solo tool still has some utility and isn't going to go away (soon), but more advanced problems need to get solved with better, more powerful tools that don't involve code that is littered with special-casing and brokenness (e.g. search) which we will never fix. Using chef-solo as a packaging tool like we do inside of EC is fine, where those recipes will never talk to a chef server and they're replacing something like an RPM postinstall script. Using chef-solo to provision your infrastructure is starting off tying one hand behind your back and will paint you into a corner on a long enough timeline, and we won't have the resources to write code to extract the people who get stuck there. Using chef-solo might be all that some people ever need, but its impossible to look at any given person deploying their infrastructure and be able to predict where they'll wind up in the future -- so we have to advise not to do this.

I don't know what to do about this project, but I'd encourage users to start playing with chef-metal / chefdk and start cutting tickets about knife solo workflow use cases that are not covered.

[matschaffer]

Thanks @lamont-granquist, I suspected that was the case (about viewing chef-solo as a dead end).

I played with chef metal a bit and I'm thinking there still could be some some value in a wrapper like knife solo to make ssh provisioning simpler.

Granted, the common driver concept is pretty solid. I think I'll start looking at what it'd take to make a knife-solo interface on top of chef-metal & chef-metal-ssh.

[lamont-granquist]

Yep, that sounds like a good idea to me...

[allaire]

@matschaffer @tmatilai Is there something similar to knife solo for chef zero? Or plan to support it? Love the fact that I can do everything via SSH (à la Ansible) for smaller projects instead of spinning a Chef server.

[matschaffer]

@allaire I think switching knife-solo to chef-zero would be great and not too difficult. But I'm mainly using ansible and puppet at the moment so haven't had much need to work on this. Could probably pick it up a bit later in the spring but I'm happy to accept contributors in the mean time. I'm available in #knife-solo on freenode for questions any time if you're interested.

[kyle-johnson]

Relevant? https://github.com/higanworks/knife-zero

[matschaffer]

Definitely. I haven't really used chef in a few years beyond maintaining this project so hard to say when I'll pick it up.

Though I have some friends using knife-solo here so I might do it for their sake when time allows.