Cookbook Caching

[AnalogJ]

Hey @tapickell @richardardrichard @berniedurfee-ge This is the followup PR to #35 and fixes #34

Basically I needed a way to generate a minimart for ~300 different cookbooks with ~4000 tagged versions. Currently minimart takes almost 40 minutes to do that, primarily because of 2 things:

• git cookbooks arn't cached, and are downloaded again every time.
• the tar.gz file for each cookbook version is regenerated each time by the web command.
• because the tar.gz file is created new each time, the AWS sync command re-uploads everything.

First I modified the minimart InventoryBuilder and LocalStore so that it keeps track of the metadata files for each cookbook that it sees in the inventory_directory. Then when a explicit_location cookbook is added via the inventory file, it checks if the cookbook already exists, and if so, does not clone and fetch the cookbook again.

To do this I had to make changes to the metadata files, .minimart.json. I added a metadata_version number to all new files because the changes I made to the matching_source? function arn't really backwards compatible. If the new metadata_version key is present, it uses the new logic, otherwise it falls back to the previous logic.

Then I modified the Web command such that you can optionally specify if you want the cookbook_directory to be cleaned on each run. By default this value is true (which is how it was before, but if you use --no-clean-cookbooks in the CLI, the Web command will check if a tar.gz already exists for a package before generating it.

After making these changes, I was able to bring down the build and deploy time to less than 2 minutes, which makes it perfect for incremental changes to a large cookbook repo.

---

Having said that, there is still a single issue left with this PR. There's a single test which continues to fail, something that I haven't figured out how to fix. I'd appreciate any help :)

[AnalogJ]

Hey guys, I could really use some direction here :)

[berniedurfee-ge]

@AnalogJ I'll take a look...

[berniedurfee]

Not sure why that test is failing... :frowning:

Looks like it should fail when you try to add a version of a cookbook to the local store that is already there, but with a different source. Maybe the test isn't being setup correctly? I'll try again later to debug...

[berniedurfee-ge]

Closing this as I've added a couple tidbits and re-pulled as #40

[berniedurfee-ge]

The fix (hope it was correct) was to create a new inventory file with a second tag for the same cookbook. This should fail, as the test returns the same cookbook with the same version. So Minimart throws an error because it means the same cookbook version was pulled in from two Git locations (different tags) which is bad.

[berniedurfee-ge]

Caching release (v1.2.0) is now in RubyGems.org...