Ability to use Tabula Rasa via Berkshelf

[nickjwebb]

Has anyone been able to run Tabula Rasa via Berkshelf? As in, point the OpsWorks stack at a custom repository and specify Tabula Rasa in the Berksfile?

I tried this today as it would allow me to run some cookbooks outside of Tabula Rasa and some inside, but due to the dependancies on OpsWorks Cookbooks, I got a bad case of dependency hell.

From metadata.rb:

depends "scm_helper"
depends "opsworks_berkshelf"
depends "opsworks_custom_cookbooks"

Resolving those dependancies, and their dependencies ends up with a huge list of basically all of the opsworks cookbooks in the form of:

cookbook 'scm_helper',
  git: 'https://github.com/aws/opsworks-cookbooks',
  branch: 'release-chef-11.10',
  rel: 'scm_helper'
cookbook 'opsworks_berkshelf',
  git: 'https://github.com/aws/opsworks-cookbooks',
  branch: 'release-chef-11.10',
  rel: 'opsworks_berkshelf'
cookbook 'opsworks_custom_cookbooks',
  git: 'https://github.com/aws/opsworks-cookbooks',
  branch: 'release-chef-11.10',
  rel: 'opsworks_custom_cookbooks'
cookbook 'opsworks_commons',
  git: 'https://github.com/aws/opsworks-cookbooks',
  branch: 'release-chef-11.10',
  rel: 'opsworks_commons'
 . . .

This is counter to the point of Tabula Rasa, as we are forced to include the apache2 cookbook from opsworks for the Tabula Rasa dependency when we wish to use the community apache2 cookbook instead.

Clearly I must be missing something... or is no one including this cookbook via Berkshelf?

[shlomoswidler]

I have used tabla_rasa via Berkshelf. What URL is your Stack pointing to?

[shlomoswidler]

Also, can you please be a bit more specific about how you want to use "some cookbooks outside of Tabula Rasa and some inside"? I'm not sure I understand what you're trying to do.

[nickjwebb]

@shlomoswidler ,

I have the stack pointing to a custom repo URL (private github repo).

As to using some cookbooks inside and outside of Tabula Rasa, I mean that I want to use some cookbooks, such as bamboo-agent (which uses chef_gem and thus fails due to https://github.com/shlomoswidler/tabula_rasa/issues/6), outside of Tabula Rasa. By "outside" I mean in the standard OpsWorks "run lists" for setup, configure, etc.

At the same time, I'd like to have some cookbooks, such as apache2, to run within the Tabula Rasa environment.

In order to include bamboo-agent I must use a Berksfile in my own repository, and thus if I also want to use Tabula Rasa, I must specify that in my Berksfile as well. However this is creating the mess noted in comment one.

It's pretty clear to me that the steps to reproduce this are not clear here. The need is also low, if we can get https://github.com/shlomoswidler/tabula_rasa/issues/6 working -- I'm going to focus attention there for now. Let's leave this be for now, if there is another need I'll create a simplified proof of concept that illustrates the issue, which may instead illustrate a mistake on my part. Feel free to close if you'd rather not have it open for a while.

[nickjwebb]

I've hit this again in a very simple test for https://github.com/shlomoswidler/tabula_rasa/issues/6.

Here is my full setup.

Custom JSON

{ 
  "tabula_rasa" : {
    "scm": {
      "type": "git",
      "repository": "git@github.com:nickjwebb/tabula-rasa-test.git"
    },
    "recipes" : { 
      "setup": [ "tabula-rasa-test::phpunit" ] 
    }
  }
}

Stack settings repo URL (this is a public one for testing): https://github.com/nickjwebb/tabula-rasa-test.git

Which has in it's Berksfile:

source "https://supermarket.chef.io"
cookbook 's3_file', '~> 2.5.1'
cookbook "tabula_rasa", git: "https://github.com/shlomoswidler/tabula_rasa.git"

I have just tabula_rasa in the custom recipes section of OpsWorks for Setup (and all other lifecycle events).

I get the same dependency error as before, when trying to run 'update_custom_cookbooks'.

[2015-06-12T00:24:43+00:00] INFO: Processing ruby_block[Move single cookbook contents into appropriate subdirectory] action run (opsworks_custom_cookbooks::checkout line 64)
[2015-06-12T00:24:43+00:00] INFO: Single cookbook detected, moving into subdirectory '/opt/aws/opsworks/current/site-cookbooks/tabula-rasa-test'
[2015-06-12T00:24:43+00:00] INFO: ruby_block[Move single cookbook contents into appropriate subdirectory] called
[2015-06-12T00:24:43+00:00] INFO: Processing opsworks_berkshelf_runner[Install berkshelf cookbooks] action berks_install (opsworks_berkshelf::install line 54)
[2015-06-12T00:24:43+00:00] INFO: Processing directory[/opt/aws/opsworks/current/berkshelf-cookbooks] action delete (/var/lib/aws/opsworks/cache.stage1/cookbooks/opsworks_berkshelf/providers/runner.rb line 2)
[2015-06-12T00:24:43+00:00] INFO: Processing ruby_block[Install the cookbooks specified in the Berksfile and their dependencies] action run (/var/lib/aws/opsworks/cache.stage1/cookbooks/opsworks_berkshelf/providers/runner.rb line 11)

================================================================================
Error executing action `run` on resource 'ruby_block[Install the cookbooks specified in the Berksfile and their dependencies]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '106'
---- Begin output of /opt/aws/opsworks/local/bin/berks vendor /opt/aws/opsworks/current/berkshelf-cookbooks ----
STDOUT: Resolving cookbook dependencies...
Fetching 'tabula_rasa' from https://github.com/shlomoswidler/tabula_rasa.git (at master)
Fetching cookbook index from https://supermarket.chef.io...
STDERR: Unable to satisfy constraints on package scm_helper, which does not exist, due to solution constraint (tabula_rasa = 0.2.5). Solution constraints that may result in a constraint on scm_helper: [(tabula_rasa = 0.2.5) -> (scm_helper >= 0.0.0)]
Missing artifacts: scm_helper,opsworks_berkshelf,opsworks_custom_cookbooks
Demand that cannot be met: (tabula_rasa = 0.2.5)
Unable to find a solution for demands: s3_file (~> 2.5.1), tabula_rasa (0.2.5)
---- End output of /opt/aws/opsworks/local/bin/berks vendor /opt/aws/opsworks/current/berkshelf-cookbooks ----
Ran /opt/aws/opsworks/local/bin/berks vendor /opt/aws/opsworks/current/berkshelf-cookbooks returned 106

I am using Amazon Linux AMI 2015.03, Berkshelf 3.1.3.

Any ideas? Not being able to run custom recipes directly from my cookbooks (including them in the Setup run list in OpsWorks) and then running cookbooks in Tabula Rasa is (potentially) slowing progress on https://github.com/shlomoswidler/tabula_rasa/issues/6.

[shlomoswidler]

I can reproduce your problem, but I don't understand why it's happening.

I can successfully include cookbooks that are hosted in S3 (a .tar.gz file). Does that work for you? Would that give you the flexibility you need?

[shlomoswidler]

Clarification: In the scenario that works for me, the cookbooks .tar.gz is NOT managed via Berkshelf - it's already been berks-ified.

Does that work for you?

[nickjwebb]

@shlomoswidler OK, yes I think the solution here is to just fork tabula rasa (already done) and put my test recipes there. Then I can call them from lifecycle events directly, outside of a tabula rasa run.

As in, I could have this specified during Setup:

tabula_rasa::my_test_recipe tabula_rasa

Thanks for s3/tar.gz suggestion, that got me thinking of how to overcome. That would work too, but I'd prefer to just use a github repo so I can iterate quickly.

[nickjwebb]

I've been successfully testing by using a fork of the tabula_rasa repo, putting the cookbook in the tabula_rasa subdirectory, and then including a custom cookbook in a different directory. Then including in OpsWorks as per usual.

I have my stack repository set to (public): https://github.com/redwireservices/tabula_rasa.git

Now I can call my gem_fix::gem_fix recipe from the run-list under setup in OpsWorks, without invoking tabula_rasa.

My Custom Chef Recipes section under Setup looks like this: gem_fix::gem_fix tabula_rasa

I think we can close this, but could we update item "1." under Usage in README.md? I think it's misleading. I'll submit a pull request.

[shlomoswidler]

I would welcome a PR. Thanks.