search

spox / batali

Light weight cookbook resolver
https://spox.github.io/batali
Other
32 stars 6 forks source link

Support for cookbook dependencies with their own 'non-Supermarket' dependencies #66

Closed kenny-evitt closed 8 years ago

kenny-evitt commented 8 years ago

I've got a Chef infrastructure repo that depends on two private cookbooks myapp and myappdev.

The myappdev cookbook depends on myapp.

The myapp cookbook itself depends on a non-Supermarket postgresql cookbook – this one by-the-way.

But when I run chef exec batali resolve -V -D from the root directory of the infrastructure repo, it seems that it can't find the non-Supermarket postgresql cookbook:

[Batali]: Loading sources... [Batali]: Loading Batali file from: /Users/kenny/@code/myinfrastructure/Batali
[DEBUG]: Cache directory to persist cookbooks: /Users/kenny/.batali/cache
complete!
[Batali]: Loading manifest file from: /Users/kenny/@code/myinfrastructure/batali.manifest
[DEBUG]: Logging units checksum for world addition. (`2c74a34f98a3dce83341bd82d7f83bba6712711af6f9aa3146f5166fd8714f76`)
[DEBUG]: Logging units checksum for world addition. (`4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945`)
[DEBUG]: Units added to world: []
[DEBUG]: Units added to world: [{"myapp":"0.1.16"}]
[DEBUG]: Logging units checksum for world addition. (`3fd8f023dbb67403bfb4a60bc977fca8a2071da8e3717dea80e2ecc36578eb3b`)
[DEBUG]: Units checksum already added to world. Skipping. (`2c74a34f98a3dce83341bd82d7f83bba6712711af6f9aa3146f5166fd8714f76`)
[DEBUG]: Units added to world: [{"myappdev":"0.1.2"}]
[Batali]: Performing single path resolution.
[Batali]: Resolving dependency constraints... [DEBUG]: Solver Unit: {"name":"~_SOLVER_UNIT_~","dependencies":["myapp (> 0)","myappdev (> 0)"],"version":"1.0.0"}
[DEBUG]: Solver world context of unit system: <Grimoire::System:70188920327140>: myapp: 0.1.16
myappdev: 0.1.2
[DEBUG]: Score <myapp:0.1.16>: 1025.9375
[DEBUG]: Score <myappdev:0.1.2>: 1025.5
[DEBUG]: Failed to locate unit: Failed to locate valid unit for: <Batali::UnitDependency type=:runtime name="postgresql" requirements="= 0.16.1">
error!
[ERROR]: Reason - Failed to generate valid path for requirements: `[<Gem::Dependency type=:runtime name="myapp" requirements="> 0">, <Gem::Dependency type=:runtime name="myappdev" requirements="> 0">]`
ERROR: Grimoire::Error::NoSolution: Failed to generate valid path for requirements: `[<Gem::Dependency type=:runtime name="myapp" requirements="> 0">, <Gem::Dependency type=:runtime name="myappdev" requirements="> 0">]`

Is it just not possible for Batali to resolve the postgresql cookbook? Or is that non-Supermarket dependencies cannot be resolved for dependencies themselves?

When I first tried resolving dependencies, the myapp cookbook did not contain its own Batali file – I was using Berkshelf previously to specify the non-Supermarket cookbook dependency. But it doesn't seem that Batali 'nests' and uses Batali files in dependency cookbooks – that would be nice if it did.

But even if Batali did resolve nested non-Supermarket dependencies, it seems like explicit support would need to be added for other dependency libraries, e.g. Berkshelf and Librarian. Neither Berkshelf or Librarian support nested non-default-source dependencies.

luckymike commented 8 years ago

I feel like what you've described should be possible, but I'm having trouble mapping it out exactly in my mind. Can you provide your Batali file (or at least the relevant portion)?

kenny-evitt commented 8 years ago

@luckymike Sure

Here's the Batali file for the myapp cookbook:

Batali.define do
  source "https://supermarket.chef.io"

  cookbook 'postgresql', '= 0.16.1', git: "git://git@github.com:phlipper/chef-postgresql.git"
end

The Batali file for the myappdev cookbook:

Batali.define do
  source "https://supermarket.chef.io"

  cookbook 'myapp',    git: "https://user:password@my-hosted-code-service-domain.example.com/Code/Repositories/Group/myapp-chef-cookbook.git"
end

The Batali file for my infrastructure repo:

Batali.define do
  source "https://supermarket.chef.io"

  cookbook 'myapp',    git: "https://user:password@my-hosted-code-service-domain.example.com/Code/Repositories/Group/myapp-chef-cookbook.git"
  cookbook 'myappdev', git: "https://user:password@my-hosted-code-service-domain.example.com/Code/Repositories/Group/myappdev-chef-cookbook.git"
end

I can't just copy the line for the postgresql cookbook from my myapp cookbook Batali file (and get resolve to work); but that seems to be because a version and a source can't be combined. Here's the output of running resolve after I did:

[Batali]: Loading sources... [Batali]: Loading Batali file from: /Users/kenny/@code/myinfrastructure/Batali
[DEBUG]: Cache directory to persist cookbooks: /Users/kenny/.batali/cache
complete!
ERROR: Gem::Requirement::BadRequirementError: Illformed requirement ["{\"git\"=>\"git://git@github.com:phlipper/chef-postgresql.git\"}"]

Is there a way to combine a source and version constraints in a Batali file?

If I remove the version constraint and re-run resolve I get a git clone error:

[Batali]: Loading sources... [Batali]: Loading Batali file from: /Users/kenny/@code/myinfrastructure/Batali
[DEBUG]: Cache directory to persist cookbooks: /Users/kenny/.batali/cache
error!
[ERROR]: Reason - git  clone '--' 'git://git@github.com:phlipper/chef-postgresql.git' '/Users/kenny/.batali/cache/git/Z2l0Oi8vZ2l0QGdpdGh1Yi5jb206cGhsaXBwZXIvY2hlZi1wb3N0Z3Jlc3FsLmdpdA=='  2>&1:Cloning into '/Users/kenny/.batali/cache/git/Z2l0Oi8vZ2l0QGdpdGh1Yi5jb206cGhsaXBwZXIvY2hlZi1wb3N0Z3Jlc3FsLmdpdA=='...
fatal: Unable to look up git@github.com:phlipper (port 9418) (nodename nor servname provided, or not known)
ERROR: Git::GitExecuteError: git  clone '--' 'git://git@github.com:phlipper/chef-postgresql.git' '/Users/kenny/.batali/cache/git/Z2l0Oi8vZ2l0QGdpdGh1Yi5jb206cGhsaXBwZXIvY2hlZi1wb3N0Z3Jlc3FsLmdpdA=='  2>&1:Cloning into '/Users/kenny/.batali/cache/git/Z2l0Oi8vZ2l0QGdpdGh1Yi5jb206cGhsaXBwZXIvY2hlZi1wb3N0Z3Jlc3FsLmdpdA=='...
fatal: Unable to look up git@github.com:phlipper (port 9418) (nodename nor servname provided, or not known)

After changing the Git URL to use HTTPS instead, resolve works:

[Batali]: Loading sources... [Batali]: Loading Batali file from: /Users/kenny/@code/myinfrastructure/Batali
[DEBUG]: Cache directory to persist cookbooks: /Users/kenny/.batali/cache
complete!
[Batali]: Loading manifest file from: /Users/kenny/@code/myinfrastructure/batali.manifest
[DEBUG]: Logging units checksum for world addition. (`2c74a34f98a3dce83341bd82d7f83bba6712711af6f9aa3146f5166fd8714f76`)
[DEBUG]: Logging units checksum for world addition. (`b3defb0688bbecb7ba941b69805ce0c04aef78e10b01d89961b7d94ba5583ab0`)
[DEBUG]: Logging units checksum for world addition. (`3f368675c81f7131bb243167725af90c2663738c720ccd373f4e34323cbb11b2`)
[DEBUG]: Units added to world: [{"apt":"2.9.1"},{"apt":"2.8.1"},{"apt":"2.7.0"},{"apt":"2.6.1"},{"apt":"2.8.2"},{"apt":"2.8.0"},{"apt":"2.9.2"},{"apt":"2.9.0"}]
[DEBUG]: Units added to world: [{"postgresql":"0.16.1"}]
[DEBUG]: Units added to world: [{"myapp":"0.1.16"}]
[DEBUG]: Logging units checksum for world addition. (`3fd8f023dbb67403bfb4a60bc977fca8a2071da8e3717dea80e2ecc36578eb3b`)
[DEBUG]: Units checksum already added to world. Skipping. (`2c74a34f98a3dce83341bd82d7f83bba6712711af6f9aa3146f5166fd8714f76`)
[DEBUG]: Units added to world: [{"myappdev":"0.1.2"}]
[DEBUG]: Units checksum already added to world. Skipping. (`b3defb0688bbecb7ba941b69805ce0c04aef78e10b01d89961b7d94ba5583ab0`)
[Batali]: Performing single path resolution.
[Batali]: Resolving dependency constraints... [DEBUG]: Solver Unit: {"name":"~_SOLVER_UNIT_~","dependencies":["myapp (> 0)","myappdev (> 0)","postgresql (> 0)"],"version":"1.0.0"}
[DEBUG]: Solver world context of unit system: <Grimoire::System:70153954119100>: apt: 2.6.1, 2.7.0, 2.8.0, 2.8.1, 2.8.2, 2.9.0, 2.9.1, 2.9.2
myapp: 0.1.16
myappdev: 0.1.2
postgresql: 0.16.1
[DEBUG]: Score <apt:2.9.2>: 90509.22222222222
[DEBUG]: Score <apt:2.9.1>: 90508.72222222222
[DEBUG]: Score <apt:2.9.0>: 90507.72222222222
[DEBUG]: Score <apt:2.8.2>: 90495.0
[DEBUG]: Score <apt:2.8.1>: 90494.5
[DEBUG]: Score <apt:2.8.0>: 90493.5
[DEBUG]: Score <apt:2.7.0>: 90475.21428571429
[DEBUG]: Score <apt:2.6.1>: 90451.83333333333
[DEBUG]: Score <postgresql:0.16.1>: 1985.0
[DEBUG]: Score <myapp:0.1.16>: 1025.9375
[DEBUG]: Score <myappdev:0.1.2>: 1025.5
[DEBUG]: Score <postgresql:0.16.1>: 1985.0
[DEBUG]: Score <apt:2.9.2>: 90509.22222222222
[DEBUG]: Score <apt:2.9.1>: 90508.72222222222
[DEBUG]: Score <apt:2.9.0>: 90507.72222222222
[DEBUG]: Score <apt:2.8.2>: 90495.0
[DEBUG]: Score <apt:2.8.1>: 90494.5
[DEBUG]: Score <apt:2.8.0>: 90493.5
[DEBUG]: Score <apt:2.7.0>: 90475.21428571429
[DEBUG]: Score <apt:2.6.1>: 90451.83333333333
complete!
[Batali]: Writing manifest... complete!
[Batali]: Ideal solution:
apt <2.9.2>
myapp <0.1.16>
myappdev <0.1.2>
postgresql <0.16.1>

I changed the postgresql dependency in the Batali file of myapp and commenting it out in the infrastructure repo. The results of running chef exec batali cache --scrub and then chef exec batali resolve -V -D:

[Batali]: Remove all contents from local cache (/Users/kenny/.batali/cache) (Y/N): Y
[Batali]: Scrubbing local cache... complete!
Batali cache information:
  Path: /Users/kenny/.batali/cache
  Size: 0.00M

and then:

[Batali]: Loading sources... [Batali]: Loading Batali file from: /Users/kenny/@code/myinfrastructure/Batali
[DEBUG]: Cache directory to persist cookbooks: /Users/kenny/.batali/cache
complete!
[Batali]: Loading manifest file from: /Users/kenny/@code/myinfrastructure/batali.manifest
[DEBUG]: Logging units checksum for world addition. (`32c0edb4d8aa6d2b8579dffaa360ac1f18e781b4690ee4919796a78417da5a80`)
[DEBUG]: Logging units checksum for world addition. (`4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945`)
[DEBUG]: Units added to world: []
[DEBUG]: Units added to world: [{"myapp":"0.1.17"}]
[DEBUG]: Logging units checksum for world addition. (`3fd8f023dbb67403bfb4a60bc977fca8a2071da8e3717dea80e2ecc36578eb3b`)
[DEBUG]: Units checksum already added to world. Skipping. (`4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945`)
[DEBUG]: Units added to world: [{"myappdev":"0.1.2"}]
[Batali]: Performing single path resolution.
[Batali]: Resolving dependency constraints... [DEBUG]: Solver Unit: {"name":"~_SOLVER_UNIT_~","dependencies":["myapp (> 0)","myappdev (> 0)"],"version":"1.0.0"}
[DEBUG]: Solver world context of unit system: <Grimoire::System:70277700883620>: myapp: 0.1.17
myappdev: 0.1.2
[DEBUG]: Score <myapp:0.1.17>: 1025941176.4705883
[DEBUG]: Score <myappdev:0.1.2>: 10255000000.0
[DEBUG]: Failed to locate unit: Failed to locate valid unit for: <Batali::UnitDependency type=:runtime name="postgresql" requirements="= 0.16.1">
error!
[ERROR]: Reason - Failed to generate valid path for requirements: `[<Gem::Dependency type=:runtime name="myapp" requirements="> 0">, <Gem::Dependency type=:runtime name="myappdev" requirements="> 0">]`
ERROR: Grimoire::Error::NoSolution: Failed to generate valid path for requirements: `[<Gem::Dependency type=:runtime name="myapp" requirements="> 0">, <Gem::Dependency type=:runtime name="myappdev" requirements="> 0">]`
kenny-evitt commented 8 years ago

The above does kind of work as a workaround, i.e. run resolve, note any Failed to locate valid unit for ... errors, find the un-located cookbook, and then add it as an explicit dependency with the non-default source info to the repo being resolved.

chrisroberts commented 8 years ago

Hi!

  1. Constraint and source cannot be combined

    When an explicit source is defined for a cookbook (path or git) constraint information is already known. This is due to the entry only being able to define a single source, and the contents of that source will include the metadata file which provides the version at that particular source. In this scenario, even if constraint information were to be provided, it would have to match the version found at the source.

  2. Non-unique cookbook names

    The source of the issue you are hitting here is really the re-use of an existing cookbook name. If we define a system with multiple disparate things all using a common name (which is essentially what is happening here), attempting correct identification via name only is not possible. This is not an issue you introduced, rather it's an issue others have introduced instead of simply using a unique naming scheme for their cookbooks. This issue is magnified within a running infrastructure where dependencies may require multiple implementations of a named cookbook (postgresql for example) and the solver cannot know which is actually correct within a particular solution.

  3. Clone error

    That looks like it was just a transient networking issue (nodename nor servname provided, or not known)

  4. Merging Batali files

    This is something that I have implemented in the past for berkshelf (via berkshelf_ext) and would be open to investigating for batali. In its current state, however, this is not a feature that is supported.

Suggestions: There are a few options you can investigate. If you are running your own chef server instance (chef server 12 or greater) you can create an organization just for cookbook asset storage. You can then use this new organization within your Batali file so it can use cookbooks within that organization when generating a solution. This is an easy way to store custom cookbook assets (internal cookbooks or customized/different cookbooks). Another option would be to run a personal supermarket to store these cookbook assets.

Outside of those, copying your customized source information to the infra repository Batali file is going to be your only option for now.

Cheers!

kenny-evitt commented 8 years ago

@chrisroberts Thanks for the detailed response.

For [1], it seems reasonable (to me) that one could treat a custom or explicit source the same way as an 'implied source', e.g. capable of providing multiple versions just as 'a supermarket'. Git tags seem perfectly suited for providing versions. I get that it doesn't work this way now, but I don't (yet) understand why it couldn't work this way potentially. Given what you've written, I'd expect to have to implement this myself. If I did tho, would you be open to merging?

[4] would be great. I actually started using Batali for this project because I thought this was already supported.

My real problem is that I'm trying to use Chef without a server. Despite it being possible, it's certainly not straightforward or easy with any existing tools (of which I'm aware). And it's actively discouraged. I persist because the Chef ecosystem is valuable and unlocking it without needing to run a Chef server seems so tantalizingly close to realizable.

luckymike commented 8 years ago

@kenny-evitt even if you don't want to use Chef Server for node data, you probably should use Chef Server or a Supermarket server as an artifact store for your cookbooks. Batali can be used to perform the dependency resolution and vendoring on the client side from either a Chef or Supermarket server.

The git option should be understood as a source "build," and as in other software contexts, you may retrieve a specific revision of the source code, but the "version" you build is determined solely by the versioning present in the revision that you've checked out.

kenny-evitt commented 8 years ago

@luckymike Thanks