Feature Request: sub-command to retrieve director manifest

[wfernandes]

Is it possible to make a command to the effect of bosh director-manifest that returns the manifest with which the bosh director was created?

Referencing https://github.com/cloudfoundry/bosh-bootloader/issues/274

[dpb587-pivotal]

This has come up a few times over the years, but we are not excited about the idea for a couple reasons...

• The only state for a create-env'd VM is the state.json. If we were to treat it as the "director database" where a manifest is normally stored, that means adding the manifest into that file. That has a couple problems though...
  • The manifest may contain credentials, which is not something the state.json file currently has. Since the state should be stored somewhere, we want to reduce the area where credentials may potentially be stored.
  • If we were able to generate the manifest to store in state.json, that means we have access to the original manifest itself. At that point though, I don't think it makes sense to try and store off a copy because it's already nearby.
• If not storing locally, that means we should store the manifest on the VM. If storing on the VM, that likely means we also need to support retrieving from the VM, too. These have a couple issues, too...
  • There is no API call for storing a raw copy of the manifest on a VM. If the goal of a VM is to be provisioned with the minimal amount of credentials/information/data it needs to be running, this would go against that philosophy. This also isn't something director-managed VMs should support.
  • Eventually users would want/need a way to retrieve their original manifest off the VM. However, this now introduces a new, significant attack vector for revealing potentially sensitive configuration data off a VM. That's not a route we want to go down, either.
• If not storing anywhere, that means a manifest must be generated or retrieved on demand. Typically, scripts should be used to document how bosh create-env is invoked, which then track the ops files and variable references in a self-documenting way. The director-manifest subcommand in that case then becomes interpolate (and without the --state option).

Ultimately what this request typically comes down to is: teams did not do a sufficient job of documenting their deployment strategies when first deploying a VM (and/or we did a poor job recommending it). Rather than provide a way to store off the manifest, we think it's better to encourage documentation/scripting of the deploy itself since it's easy to flag or swap out create-env to review the manifest.

So, those are some of the typical discussions we've seen around this. Does that explain things a bit more? Do you have other thoughts or perspective we should consider around this?

[dpb587-pivotal]

:link: cloudfoundry#bbl-users (2017-02-28T14:10:44)

Maybe I should ask bosh to have the director store its own manifest or something.

:link: cloudfoundry#bosh (2017-12-15T08:04:05)

Is there a command I can run to get the manifest that was used to create-env the director, from the director? Like an equivalent to bosh -d <depl-name> manifest but for the director's manifest?

:link: cloudfoundry.slack.com#bosh (2017-07-28T12:19:27)

How can I download the director manifest for an existing environment?

:link: cloudfoundry.slack.com#bosh (2017-05-22T17:03:36)

I need the commands to download the bosh manifest

[evanfarrar]

There is no API call for storing a raw copy of the manifest on a VM. If the goal of a VM is to be provisioned with the minimal amount of credentials/information/data it needs to be running, this would go against that philosophy. This also isn't something director-managed VMs should support.

I thought generic configs was an api for storing arbitrary YML, surely the bosh cli could use that? I realize that this is not the canonical place, but if the bosh cli uploaded the secret-free YML of the director manifest after creating the director with create-env, then it would at least be a handy place for future generations of operators to answer questions like "wait, what options did we enable on this director, anyway?"

Ultimately what this request typically comes down to is: teams did not do a sufficient job of documenting their deployment strategies when first deploying a VM (and/or we did a poor job recommending it).

If BOSH has developed a good convention for structuring director manifests, and directory trees of many of those manifests, then why is the BOSH cli not aware of what it is? Can we enshrine this convention in code? What if bosh envs and bosh env were aware of this structure, and could tell you everything about that director?

[dpb587-pivotal]

... generic configs ...

True, although keep in mind that create-env is generic and doesn't only deploy the director. People would also be interested in this functionality for concourse or other deployments.

This would also introduce a dependency for create-env to know how to authenticate to a director and assume it's a recent director which supports generic configs. I could see this as a followup step if people really really really wanted to store off their manifest as a config... but I don't think that should be tied to create-env.

... directory trees ...

Not sure what you mean. I do think it might be useful to have a few more conventions around recommended directory layouts of manifests and ops, but I'm not sure the create-env command would ever be create-env this-directory-of-ops-n-vars-n-stuff. Or, maybe with a lot of conventions, but still, I imagine a file which lists manifest, ops/vars references to be more explicit. This is kind of similar to discussions in https://github.com/cloudfoundry/bosh-cli/issues/287.

What you're mentioning kind of sounds like the Configured with ./configure --so-many --opts which shows up in some binaries' --version-type output. Even in this case though, it requires a lot of context about what those files represent (maybe my super-fast-director-ops.yml means something different than another environment).

@tjvman says "create-env creates parity and reasonable expectations with env/envs command"

I hadn't really considered in a while how the generic bosh create-env is misleading if you end up doing bosh -h and randomly discovering envs. I think there was some lengthy discussions a long time ago when this was introduced, and the confusions it implies since env/envs are director-specific commands. The later points you mention kind of stem around that, and in that sense, I agree.

Generically speaking, I'd expect bosh envs to tell me about the "envs that I created". In that sense, they wouldn't care whether they're bosh, concourse, vpn, nat, whatever. One of the pieces of information it could tell me might be how it was deployed, or which directory I was in, or other highly-specific, local information. Seems a bit weird and too detailed, but crazy possibility.

If that were a direction which would be desirable, I'd then expect the ~/.bosh/config to have many more details, not just bosh director auth/ca information. So, if starting from that point... then there would need to be a command which "binds" a generic "env" which was created to the "director" functionality that the rest of the bosh CLI is used to. So, maybe I create-env (with a -e alias?) and then follow up with a set-director-env where I might pass a CA cert and which toggles the env as being director-friendly or something.

Really though, that sounds more like a separate bootstrap, wrapping tool. But that's more reminiscent of the dedicated, separate bosh-inti command.

Dunno, just brainstorming; hadn't really thought much about it recently. I appreciate the continued discussion.

[evanfarrar]

True, although keep in mind that create-env is generic and doesn't only deploy the director...

@tjvman https://github.com/tjvman says "create-env creates parity and reasonable expectations with env/envs

Aha! I think we might be on to something! I don't particularly find create-env particularly useful outside of making a director, and I don't think even concourse-deployment is that strongly advocating using it as the best way to make a concourse. Could we rename create-env "deploy-without-a-director"? and begin to make "create-env" a bit more opinionated about how state and credentials are stored? That way, targeting and reasoning about directors could be more foolproof by relying on these conventions?

[damzog]

Hi, I find this discussion very interesting.

Let me tell what we are struggling with: We have created a script and quite some vars and ops files to create our director with create-env. Everything is checked into vcs including the state.json (excluding creds.yml) reflecting the state of our director. Unfortunately if some operator changes config and re-executes deployment on his personal clone of our config repository state.json will be changed. If this operator now forgets to checkin state.json and another operator changes config in his personal folder he will most probably work on an outdated state.json ...

Now coming to a possible solution: As I understand create-env will always create only one vm. This vm is based on normal stemcell. So why not store all state (manifest and state.json) on this vm? bosh cli could then be pointed to that vm and operate similar as a normal director would do - at least make it possible to download manifest and state.json. Does that make sense?

[cppforlife]

So why not store all state (manifest and state.json) on this vm?

this is one of these tough bootstrapping decisions... if we encourage people to keep their state in the vm, it's very likely that at some point that vm will be damaged and there will be no easy way to get that state out. that means that "recovery" procedure (which is just re-running bosh create-env) will become fairly complex and will be no longer exercised on regular basis.

[bosh-admin-bot]

This issue was marked as Stale because it has been open for 21 days without any activity. If no activity takes place in the coming 7 days it will automatically be close. To prevent this from happening remove the Stale label or comment below.