Package manager

[karellm]

Hi,

I'm looking into the different provisioning systems and I really like Sprinkle. After giving a shot to Chef and Sunzi, Sprinkle actually feel like the right balance between features, complexity and support.

However, it seems to lack a package manager at first glance. Is there anything like librarian-chef (https://github.com/applicationsonline/librarian) or remote recipes (https://github.com/kenn/sunzi#remote-recipes) in Sprinkle?

Thanks,

[joshgoebel]

You can find a lot of git, but I think really you're supposed to build your own. I mean most common things have OS level packages so a package is really just apt "blah"... might be neat to find a nice way to expose the distro specific installers from the recipes directly without having to create single line packages.

[joshgoebel]

Or by package manager you mean a remote list of packages you can access - no nothing like that.

[karellm]

I indeed meant an equivalent of bundler for packages. You can provide the git repo, ref and it will take care of compiling your /packages directory for you.

Unrelated question: I find the documentation very poor (well inexistent...). Thankfully there are some good blog posts out there but am I missing something? For instance I can't find anything on how to deploy without Capistrano.

Thanks for your help.

[joshgoebel]

There are great comments in the source... read the source or generated rdoc. Raw SSH works quite well. Examples in the comments/rdoc as I said already.

Thanks, Josh Goebel

On Sep 11, 2012, at 3:26 PM, Karel Ledru-Mathé notifications@github.com wrote:

I indeed meant an equivalent of bundler for packages. You can provide the git repo, ref and it will take care of compiling your /packages directory for you.

Unrelated question: I find the documentation very poor (well inexistent...). Thankfully there are some good blog posts out there but am I missing something? For instance I can't find anything on how to deploy without Capistrano.

Thanks for your help.

— Reply to this email directly or view it on GitHub.

[joshgoebel]

There has been talk of an official package repository, but not anything like a Gemfile where you could just pull packages from all over... I don't think that type of approach works will without static sources anyways... I'd hate to go to bundle install and suddenly all the sprinkles I linked to aren't there anymore. And if you're just fetching them once and then sticking them into packages a much simpler approach would probably work.

[joshgoebel]

Did you see the example directory?

[joshgoebel]

Another issue is multiple OSes... i"m not sure we should have ubuntu packages right next to OpenBSD packages all in one huge repository...

[stuartellis]

FWIW, I started putting together a repository of reusable packages, and found that almost all packages were OS-specific, so quickly ended up creating a "debian" directory for Debian/Ubuntu and a "generic" directory for the cross-platform ones. I also thought about having a separate directory for source installation packages, but that's not something I usually do myself.

[joshgoebel]

I imagine if we go that route we'd have a sprinkle-packages gem... but when you included it what would you get? anything? Or would you have to then require 'debian/web/apache' or such? I think it'd get slow loading and parsing 100s of packages.

[ghost]

I'm using Sprinkle heavily in more than one platform.

I also need a place where packages and configurations can be shared, so I had the idea was to organize all my recipes (and others I took from Internet) on a single GitHub repository and build a Web interface on top of Sinatra to generate customized scripts (pretty much the same way JQuery UI allows me to build a custom .js package).

Now that I see this discussion, I created the repository to tell you a bit more about my idea. Hopefully you'll find it useful and give me comments.

https://github.com/hagarelvikingo/sprinkle-packager

Regards,

Rodrigo.

[stuartellis]

@yyyc514 - I hadn't thought about gem packaging before today, but as I mentioned on issue #48, my first thought is to deliberately avoid the question by just providing a method for people that want gems to build their own.

@hagarelvikingo - I also thought about this after tinkering with a system that involved reusable packages, and filed issue #48 a long time ago to see if there was any interest. @yyyc514 replied to this recently and we've been discussing it more on that issue.

My incomplete system was this:

https://github.com/stuartellis/spritz

This was intended as a template Sprinkle project for people to fork and use on their deployment machines, rather than a central repository, but I quickly hit issues about how to organize the packages.

[stuartellis]

I ought to say that I'm happy to contribute to whichever the "right" repository is, however it is maintained.

@hagarelvikingo - I've read your README, and it sounds really good.

One thought - having a searchable system ideally requires some simple convention for putting metadata inside the package files. Fixing properties like category ("database") in the directory names might cause issues later. Maybe even the OS version shouldn't be there. For example, many packages won't be specific to a version of the OS, and new Ubuntu versions will keep coming every six months...

[ghost]

@stuartellis you're right. But I think nobody will want to build a database. Perhaps some rubydoc-style or jsonified documentation on what it provides, what it requires, which version is it and what package manager it supports. Thank you for your insights.

[stuartellis]

@hagarelvikingo - I agree. That's what I was thinking, something like a hash or structured comment header would be enough.

[joshgoebel]

I've been thinking about this... I'm not sure a package manager is what users really want/need. It seems to me most of the time someone is really more interested in policies... I want a "rails server"... or a "webmail box"... or a "webserver". And you see this in the github repositories... they usually contain a variety of packages that when all combined together provide the necessary functionality. Often they include their own sprinkle script, Capistrano file, etc...

I don't think finding these best of breed examples and sticking them all in place is the right idea. After all what is "rails"? Sure we could have package :rails; gem "rails; end but what does that really accomplish that the user could not do for themselves? And if someone really wants a rails stack, that's outside the scope of mere package management - you have thin, unicorn, passenger, etc... not to mention 1000 ways to setup and configure ruby (source, apt, rvm, rbenv, etc)... better to let the policy maintainer maintain their own specific flavor that users can consciously decide to pull from.

So instead of "it seems to lack a package manager at first glance" I'd like to hear more specific examples of what is missing... and if those specific examples are more in terms of policies than packages then a package manager is not the problem here.

It seems a package database would only be really helpful for source packages with non-typical installations, or am I missing something here?

[ghost]

@yyyc514 in fact I don't want to describe a package manager but a policy builder. And in fact, I think it's not a problem that sprinkle should solve, because overcomplicating what it does will degrade its performance. I will describe the scenario I am trying to solve.

If I want a Rails stack, a package manager will typically tell me "Hey, wanna build Rails? You may want a webserver, a database, a Ruby version manager and a Ruby flavour." That is enough to describe just one role, and for that, the examples directory is enough.

Now, building a policy for a second role shouldn't be difficult either: let's say I want the database to stay in another machine. But there is a bunch of stuff someone may (and eventually will) forget. And so on with the third (mail), fourth (caching), fifth (monitoring) and twentieth role (staging). The examples directory will fall short.

Move on to a separate repository for examples: which web server to choose? how to install it? how to update? how to... whatever? That will quickly become a mess! That's what happened to @stuartellis when proposing a "forkable" (if such a word exists in english) repository.

An option is to keep everyone creating recipes with their own preferences and publishing in their own repositories, making their own lives difficult because they now have to maintain their own scripts, and making our own lives difficult because we now have to use Google to find recipes that probably aren't that revised.

Another one should be: to get back to a centralized repository but add a small metadata (it doesn't need to be as complex as apt-get), a set of good practices on writing packages so that they can meet a reasonably good amount of sprinkle users and an interface to build a skeleton which allows users to build their own starter scripts. Sprinklers will still be able to build their own scripts.

Again, that's not something that sprinkle should solve, but a task for something that uses sprinkle without requiring it to be modified.

[stuartellis]

Agree again. I don't think that adding a package manager as such is necessary. Technically, I'd just like to be able to have an obvious, supported way to draw packages from an external repository, and a default public external repository.

My own interest is enabling people to do only as much work as is necessary. In particular I'd like new users to be able to get a quick success for a completely standard stack (Ubuntu, MySQL, Passenger, Rails), and then be able to tweak and expand what they have, rather than having to build from scratch.

Ideally, I'd like to present them with a repository of packages for common things that follow some explicit conventions, and a sensible reusable project that builds on it, with some policies for common problems and an example runner.

They could then move on to building their own policies from the standard packages, then making their own packages and so on, diverging as their needs and expertise grow. And, since they've started from a common base, they would be able to not just contribute back to the default repository (if it's still useful to them), but also would have some common ground for sharing and conversations with other users.

[joshgoebel]

Yeah at this point I'm more interesting in seeing what you guys come up with and perhaps adding hooks to sprinkle (if any are even needed). Seems the best way to do this would be gems and normal ruby requires.

gem 'sprinkle-rails'
require 'rails/packages/all'

Sprinkle::Rails.add_rails_policy ...

I guess it's the last line there that has me wondering about hooks... if a gem defines policies... should you just be able to pull them directly from the gem... but then that's mighty hard to customize anything at all.

Or a more custom build:

gem 'sprinkle-rails'
require 'rails/packages/db'
require 'rails/packages/web'

policy ...

[stuartellis]

I'm personally not in favour of using gems for packages, although I'm not against it as an option. It seems a bit inflexible and could bring up extra issues like how to divide up the packages and namespace pollution (if you use the public RubyGems server), as discussed on the other issue. The only big advantage that I can think of is that it provides versioning without needing to add it to Sprinkle itself.

What I would ideally like is something like this:

# Use http://packages.sprinkle-tool.com/<operating-system>,
# or your own server
# Set to 'none' or empty to not use a package repository
package_repository 'http://packages.sprinkle-tool.com/debian'

# Default directory for packages
# Use require to get packages from other locations
# package_dir 'packages'

# These packages will automatically be required.
# If they are not in the directory specified by package_dir
# and there is a package_repository, they will be downloaded.
packages :git, :curl, :mysql, :rvm_current_user, :passenger_nginx

policy ...

[joshgoebel]

I don't think that really recognizes what I said earlier, that what most people want to share aren't really packages per se, but rather policies... that's what you see on github AFAIK. People sharing full stacks, policies. Clone and tweak. In fact packages often aren't helpful unless part of policies... we don't want a ton of packages that are just:

apt "package_name", :dependencies => true

There is no value in maintaining a huge list of those... and you really don't have the context to configure a package in a lot of cases without knowing the larger picture (policy). And I'm not sure sprinkle should get in the habit or installing a package with the default package management tool of an OS only to tweak the settings in some weird way.

Again feel free to build whatever, but I'm not sure this is what users want or need. I think what would help a lot:

• Ability to easily pull a policy (or multiple policies) from github and turn it into a working sprinkle setup:

cd my_new_shiney_server
sprinkle-policy fetch https://github.com/rails/sprinkle-rails
sprinkle-policy fetch https://github.com/grasshopper/sprinkle-mongo
sprinkle-policy fetch https://github.com/grasshopper/sprinkle-memcache
vim setup.sprinkle

In other words just an easier way to manage what people are already doing. Thoughts?