Newer chef-dk version cause karamel to not work for complex cluster defns

[jimdowling]

In the hopsworks-aws.xml example, we have a DAG of dependencies with the hopsworks cookbook being at the top. It is dependent on 14 other cookbooks. Karamel generates a solo.rb file with references to all of the cloned cookbook directories - like this file_cache_path "/tmp/chef-solo" cookbook_path ["/tmp/cookbooks/kafka-cookbook/berks-cookbooks", "/tmp/cookbooks/epipe-chef/berks-cookbooks", "/tmp/cookbooks/kzookeeper/berks-cookbooks", "/tmp/cookbooks/kagent-chef/berks-cookbooks", "/tmp/cookbooks/ndb-chef/berks-cookbooks", "/tmp/cookbooks/dr-elephant-chef/berks-cookbooks", "/tmp/cookbooks/hopsworks-chef/berks-cookbooks", "/tmp/cookbooks/zeppelin-chef/berks-cookbooks", "/tmp/cookbooks/hops-hadoop-chef/berks-cookbooks", "/tmp/cookbooks/flink-chef/berks-cookbooks", "/tmp/cookbooks/livy-chef/berks-cookbooks", "/tmp/cookbooks/elasticsearch-chef/berks-cookbooks", "/tmp/cookbooks/spark-chef/berks-cookbooks"]

As hopsworks is the root of the DAG, all cookbooks will be found in cloned hopsworks path. So, for example, flink is found in the hopsworks-chef/berks-cookbooks folder above. But it's also found in the flink-chef/berks-cookbooks folder. Newer versions of chef-solo don't like this scheme. They fail.

[jimdowling]

I backed out the commit upgrading the chef-dk version until we solve this problem. I have an idea about how to solve the problem but @kamalhakim we need to discuss it first.

[kamalhakim]

Oj.. it took me a while to introduce this flexibility... you could easily change the default version number ot 0.8.0 or something instead of rolling back everything..

[jimdowling]

We can cherry-pick the commit back in again when we fix the problem. The fix will require some re-factoring, i'm sure. I tried the following for the hopsworks cluster defn.

Include only 1 cookbook: hopsworks (root cookbook with all dependencies in Berkshelf). Add all the recipe names for all dependent cookbooks in the metadata.rb for hopsworks. Add all the attributes for all dependent cookbooks in the metadata.rb for hopsworks. However, i assume there are assumptions about cookbooks -> recipes. This kind of solution would mean there would only be one path in the solo.rb file. All cookbooks would be there, only 1 would need to be cloned. I have a script for copying attributes from dependent cookbooks into the metadata.rb file for hopsworks. A better soln would be to parse the Berksfile and build the dependency graph from it.

[kamalhakim]

Hi @jimdowling

Based on what we discussed and the fact that a cluster definition might have more than one root cookbook, it is necessary to address the issue of running "berks vendor" for more than one cookbook but in the same repo.

The solution i can think of is creating a dummy cookbook and adding all the root cookbooks from cluster definition as dependency into the dymmy/Berksfile and then vendor everything in one go.

That being said i tried it for dymmy->hopsworks dependency and I get the following error, do you have any idea? Is it because we must mention a specific version number of "dela" in hopsworks?

ERROR: "berks vendor" was called with arguments ["cookbooks", "--verbose"] Usage: "berks vendor [PATH]" root@precise64:/home/vagrant/dummy# berks vendor cookbooks
Resolving cookbook dependencies... Fetching 'dummy' from source at . Fetching 'hopsworks' from git://github.com/hopshadoop/hopsworks-chef.git (at master) Fetching cookbook index from https://supermarket.chef.io... Unable to satisfy constraints on package dela, which does not exist, due to solution constraint (hopsworks = 0.1.0). Solution constraints that may result in a constraint on dela: [(hopsworks = 0.1.0) -> (dela >= 0.0.0)] Missing artifacts: dela Demand that cannot be met: (hopsworks = 0.1.0) Unable to find a solution for demands: dummy (0.1.0), hopsworks (0.1.0)

[jimdowling]

An alternative would be to do git clone as we currently do for all root cookbooks, then run 'berks install'. I think I installs all cookbooks to ~/.berkshelf/... or ~/.chef

[jimdowling]

Then include that dir in the solo.rb file. I don't know how it deals withe different versions of cookbooks though

[kamalhakim]

It seems "berks install" is deprecated in new chefdk or something it suggests to run "berks vendor".

Yes different version is another issues i guess.

[jimdowling]

Berks vendor doesn't care about versions