support for incremental updates to inventory

[AnalogJ]

We have a few hundred cookbooks stored in git (with almost 4000 tagged versions) which minimart handles pretty easily.

We manage our cookbooks using a continuous deployment pipeline which tests, version bumps, tags, commits to git and uploads our cookbooks to a chef-server automatically on pull requests. We're trying to replace our chef-server/berkshelf-api with minimart deployed to s3.

It would be amazing if we could just kick off a minimart mirror command after updating the inventory.yml file. The problem is that the mirror command seems to fetch all the cookbooks again, even if they already exist in the inventory/cache. With ~4000 cookbooks, this means we can't just replace our chef-server upload step with a minimart deploy.

How difficult would it be to have minimart check for the presence of cookbooks in the inventory directory, and then skip them during the fetch phase?

[richardardrichard]

If you are pulling cookbooks from a source like the supermarket then Minimart shouldn't be fetching the same version of the same cookbook more than once (if I remember correctly). If you are using Git as a source, then we are fetching the same cookbook every time, just because the commits could have been made to that branch/tag (I'm pretty sure Berkshelf does the same thing). It might be possible to either add a flag to turn that off (which is not enabled by default), or run some series of Git commands to determine that the local SHA matches the remote one on that branch (we currently blow away the .git directory once we are done cloning the repo/checking out the branch we need, etc....)

Anyways, really just mentioning how it currently works, but that could definitely be annoying.

[AnalogJ]

I can create a PR if you can point me in the right direction.

It looks like inventory_builder.rb already checks for existing cookbooks in the local_store, which I assume isn't populated by git cloned cookbooks?

I checked the contents of the .minimart.yml file which seems to be defined in git_requirement.rb. What do you think about adding a new ref property in addition to the commitish property in the .minimart.yml file, only for branches and tags? Basically I'm thinking we can verify that nothing has changed after cloning the repo by testing the repo tag ref against the inventory .minimart.yml ref