Implement Hiera lookups of recipe data from templated YAML files

[tomeon]

Phew! A mere one year and several weeks after #103, I'm back with a Hiera-based implementation of recipe data lookups. This is a big PR; here's an overview of the changes:

FPM::Cookery::Hiera

• Added an Instance class that subclasses ::Hiera, wrapping a Recipe class to do do lookups in YAML (and also JSON) backends. Also, a supporting Defaults class that provides sane defaults for the options to the Hiera contstructor.
• Added a Scope class that uses Recipe methods, Facts, and ::Facter for %{scope("...")} interpolations in Hiera data files.

  FPM::Cookery::Log

• A Hiera class that passes through the hiera gem's messages to the configured FPM::Cookery logger. This class is aliased to ::Hiera::Fpm_cookery_logger for reasons explained in lib/hiera/fpm_cookery_logger.rb.

  FPM::Cookery::Book

• Added the inject_class_methods! method for defining the filename and config methods on recipe classes during FPM::Cookery::Book.load_recipe. This is done so that the hiera and lookup methods are available for use while defining Recipe subclasses.

  FPM::Cookery::CLI

• Added --data-dir for specifying the directory where Hiera data files are located.
• Added --hiera-config for specifying the path to an (optional) Hiera configuration file.

  FPM::Cookery::Config

• Added data_dir and hiera_config attributes corresponding to the --data-dir and --hiera-config CLI options.

  FPM::Cookery::InheritableAttr

• Factored out FPM::Cookery::BaseRecipe.attr_rw and FPM::Cookery::BaseRecipe.attr_rw_list into a separate module, adding attr_rw_hash and attr_rw_path.
• Module provides class methods that return lists of defined attributes; these methods are used with FPM::Cookery::Recipe.apply to do Hiera lookups on all recipe attributes.

  FPM::Cookery::Environment

• Made Environment a subclass of Hash to facilitate easier cloning on inheritance through InheritableAttr.

  FPM::Cookery::Recipe

• Added .hiera class method that represents a Hiera::Instance object
• Added .lookup class method that delegates to the hiera object's #lookup method.
• Restructured #initialize to reflect the fact that .filename and .config are now class methods.
• Made a number of methods that were previously instance methods (such as all of the filepath-related methods) into class methods so that they can be used in Hiera %{scope("...")} lookups at class definition time.

  FPM::Cookery::Source

• Added the #fetchable? method that returns false if @url.empty? and true otherwise.

  FPM::Cookery::SourceHandler

• Added the #fetchable? method that delegates to the @source instance variable. This method is used in order to determine whether to enter the source-downloading block in FPM::Cookery::Packager#dispense. This corrects an issue I had introduced by making classes like FPM::Cookery::RubyGemRecipe inherit from Recipe rather than BaseRecipe: Package#dispense would raise an exception when trying to fetch a recipe.source that returned nil.

  Miscellaneous

• Added pry as a development dependency for a nicer REPL when manually testing code
• Added simplecov as a development dependency, and increased some test coverage for classes and modules that weren't otherwise affected by this PR.
• Removed exit calls from library code and replaced with FPM::Cookery::Error clases that are rescued by FPM::Cookery::CLI. The exits were causing my REPL to mysteriously quit during manual testing.
• Added lots of tests!

I also have additions for the wiki that provide documentation of public methods of interest. It appears there's no easy way to submit wiki PRs, though :(. While I figure something out, here's the Hiera page, and the updated Recipe Specification page.

Thanks in advance for considering this PR!

[boc-tothefuture]

This is pretty cool @BaxterStockman !

Two things I noticed while looking at it. (1) Should datadir be data_dir? The command line has it as --data-dir which seems inconsistent. (2) Is there a full example to look at? The wiki page indicates it points to one for redis.. But the item linked looks like a standard recipe to me?

Great work, btw! I think this will really lower the learning curve for people.

[tomeon]

Thanks, @brian-oconnell!

To point (1), agreed that it's a bit off, but I believe it follows the same convention as, e.g., the pkg directory, which is --pkg-dir on the command line, pkg_dir in config.rb, and pkgdir in recipe.rb. As to (2), I should have noted in my initial PR comment that the link is to the master branch of this repository and therefore doesn't point at the moment to a recipe that demonstrates Hiera in action. However, the new redis recipe and its supporting .yaml config files are viewable in the attached changeset.

Thanks again, and I too hope that this will make it even easier to cook packages with fpm-cookery! :)

[bernd]

Wow, thanks! I hope I will get to review and test this soon.

[beddari]

Yay, thanks for taking a shot at this!

[boc-tothefuture]

@BaxterStockman Trying to test out changes in your branch here.. It looks like you have simplecov as a regular dependency not development dependency. Can this be a development dependency? It broke my install of your version of the gem.

[tomeon]

@brian-oconnell -- thanks for catching that; it was an oversight caused by cut-n-paste from a gemspec for a gem that really does require simplecov at runtime. I will pull it out ASAP.

[af6140]

When will we merge this pull request? externalizing recipe data is sweet.

[bernd]

I built some packages and there doesn't seem to be any breakage so far. I will merge this now and fix any issues later, if there are any.

Thank you very much for the PR! :smile:

I also have additions for the wiki that provide documentation of public methods of interest. It appears there's no easy way to submit wiki PRs, though :(

I started to write documentation in this repo which is published on readthedocs. See https://github.com/bernd/fpm-cookery#documentation for details.

[davewongillies]

@BaxterStockman :pray: I'd love some documentation please! I'm confused as to how hiera looks up different variables like in the example redis recipe and the existing documentation gives me no clues

[tomeon]

@davewongillies -- apologies for the late reply. I have been meaning to get around to translating the wiki docs I mentioned in the initial PR comment into reStructuredText. I expect to knock this out soon now that the holiday season is near.