Add support for specifying a name during bosh create release

duglin commented 10 years ago

For our development process we take CF and add some additional tweaks based on our specific needs, from that we then build a new IBM specific release via the: bosh create release ... command. However, since we don't want to create a "final" release we end up creating a "dev" release. This is ok, however, the name of the release isn't what we want it to be. We want it to follow our own naming pattern based on our process and needs. In order to achieve this we have a real funky step in the process that opens up the release.MF file for each tar ball created, modifies the "name" and "version" fields and then repackages it back into the tar ball - then we can upload it to the director.

It would be really nice if there were an option on the "bosh create release" command to specify the name and version # so we could avoid this hack.

We'd like to work on the PR to do this, but before we started we wanted to bring up the topic to see if there would be any objection to this new feature, or even if there's a way to get what we want using an existing feature that we didn't know about.

cf-gitbot commented 10 years ago

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/71283492. This repo is managed by the 'BOSH' team.

bosh-ci-push-pull commented 10 years ago

Hi Doug:

That's perfect (bring up a discussion before pouring effort into a PR). The BOSH team and I have actually been giving a lot of thought to releases, dev/final, versioning, etc. I'd encourage you to first have a look at the stories in this epic: https://www.pivotaltracker.com/epic/show/1187352 - "Release Versioning". Then, let's have a conversation about where you see gaps, whether we are going in the direction that works for you, etc. We've sought a lot of customer/downstream input.

Looking forward to next steps

Greg

Greg Oehmen Cloud Foundry Product Manager - Bosh Pivotal

On Tue, May 13, 2014 at 5:13 PM, cf-gitbot notifications@github.com wrote:

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/71283492. This repo is managed by the 'BOSH' team.

— Reply to this email directly or view it on GitHubhttps://github.com/cloudfoundry/bosh/issues/578#issuecomment-43028796 .

duglin commented 10 years ago

While I don't get a 404 for that link, I only see the title, no content. Do I need more ACLs? My Pivotal ID is "dougdavis1".

bosh-ci-push-pull commented 10 years ago

That link should be public but let me verify.

On Tuesday, May 13, 2014, Doug Davis notifications@github.com wrote:

While I don't get a 404 for that link, I only see the title, no content. Do I need more ACLs? My Pivotal ID is "dougdavis1".

— Reply to this email directly or view it on GitHubhttps://github.com/cloudfoundry/bosh/issues/578#issuecomment-43030178 .

Greg Oehmen Cloud Foundry Product Manager - Bosh Pivotal

duglin commented 10 years ago

ok my bad, I see now that the "epic" is supposed to be empty, I didn't notice the link on the right to the stories under it.

From the list, it appears that 62085266 might be one that's closest to what I'm looking for, but it seems to only focus on "final" releases. And, I need to apologize first, but its been a while since I put in this hack to tweak the release name and so my memory of dev vs final releases is probably a bit blurred. But, I seem to recall that "final" releases of CF should really only be done by Pivotal. And that would then leave the rest of the community with just "dev" releases. I seem to recall this having to do with version numbering and such.

However, perhaps with 62085266 we can pretty much ignore that rule and simply give our final release any version we want, even one that is totally different than the CF version we're basing it on. Is that the intent? If so, that's good, however, it would then be nice to also allow us to set the name of the release.

Let me give some details of our naming scheme/process:

we grab cf v170 from github/cf-release
we add our tweaks and extensions to the src
bosh create release --with-tarball --force
we rename the release to be: ibm-v170.x where "x" is our Xth version of 170 that we're building/deloying

So, I think the --version flag in 62085266 will cover the 170.x part, but not sure how we'd get the ibm-v part.

I know this is probably off topic, but let me ask, it seems to me that BOSH is pushing people towards a particular build model. By that I mean it has knowledge of previous builds and version numbers (as mentioned in 62085266 when it talks about comparing version numbers numerically) and tries to use that particular build patten in its processing, and I can understand that to some extent. However, that makes it harder for folks (like us) who do not necessarily have those same requirements. For example, every build we do is basically 100% independent of any previous build so the version number, or even the entire version string, is just a string to us. We don't really care about the split between dev build and final build, they're all just builds and whether its a 'dev' or 'final' is really more dictated by what we choose to do with it. For example, a 'dev' build might end up being a 'final' build if it passes all of our test. And it would be up to our CI/CD pipeline to decide when to increment the major version number instead of the minor one. So, I guess my question is whether you guys have considered modifying things such that there's just the option of just having a "build" w/o preset notion of dev vs final? Or is that totally weird given how things are designed?

stupakov commented 10 years ago

@goehmen Could you please address this when you have a chance?

Thanks, CF Community Pair (@d, @stupakov)

d commented 10 years ago

@duglin Hi Doug, if you're creating dev releases, you do have a chance to name them (to differ from the final release names). The first time you create a dev release, the CLI will prompt you for a dev release name, FYI.

Jesse

duglin commented 10 years ago

hmm, are you able to give it a name on the command line so it can be automated and can I give it a version number - like 171.1 ?

maxbrunsfeld commented 10 years ago

Hi Doug, Ah you cannot specify a name (say with a flag) without hacking dev.yml. @goehmen Do we have such a feature planned?

Thanks, CF Community Pair Jesse and Max (@d, @maxbrunsfeld)

bosh-ci-push-pull commented 10 years ago

Yes. See this story for details: https://www.pivotaltracker.com/story/show/62085266

Note that the story was created Dec 6, 2013. Best to pick up the commentary starting on May 1, 2014. There is a lot of detail there. I'd love to hear from folks whether this story (and the epic that it is part of) are solving release development/deployment issues and making releases better and easier.

Best

Greg

Greg Oehmen Cloud Foundry Product Manager - Bosh Pivotal

On Tue, May 27, 2014 at 10:37 AM, maxbrunsfeld notifications@github.comwrote:

Hi Doug, Ah you cannot specify a name (say with a flag) without hacking dev.yml. @goehmen https://github.com/goehmen Do we have such a feature planned?

Thanks, CF Community Pair Jesse and Max (@d https://github.com/d, @maxbrunsfeldhttps://github.com/maxbrunsfeld )

— Reply to this email directly or view it on GitHubhttps://github.com/cloudfoundry/bosh/issues/578#issuecomment-44308597 .

duglin commented 10 years ago

I think Adam's comment:

Regarding auto-incrementing of the "minor" version, I think that's a mistake. Bosh can't enforce semantic versioning because it can only guess at the semantics of the release. I would prefer the release developer always specifies a version number but in the absence of that I'd rather not have bosh apply semantics when it doesn't know how.

comes closest to my thinking on this.

And, I'm sure I'm just being dense here, but I'm still confused as to why there is a distinction between the various types of releases instead of just "build it" and let the release manager decide what to call it and how to use it (dev vs final vs hotfix vs just-because). If there are functional differences (meaning, certain actions are taken during a "final" build but not during a "dev" build) then why not just expose those as flags (--do-xxx) and let the release manager decide when to invoke those steps. I'm not sure why those actions have to be tightly linked with a version string. Just feels like BOSH is trying to be "too smart" and I'd prefer a "dumber" make-like feature that I can script around based on my CI/CD process needs rather than trying to make my process fit into BOSH's notion of how I should work. Or at least, make those "smarts" optional so I can do my own "smarts" when needed and BOSH doesn't get in my way, but rather just does what I tell it to do.

karlkfi commented 10 years ago

My understanding of the current cli version behavior:

by default the manifest version is used
the --version flag allows setting a new version to use
the --final flag either uses the specified final version (via manifest or version flag) or truncates the "#-dev" suffix and auto-increments
the --rebase ignores the specified "#-dev" suffix and replaces it with a new one auto-incremented from the latest (with the same final version) from the director db.

My current work on user supplied versions should allow all of the above to continue working in a reverse compatible way for single number final versions, but operate subtly different if a more complex user-specified version is used.

It could be argued that --final behavior is confusing in the context of user-specified behavior, but I'm hesitant to remove it after a single pass. I'd rather get more feedback on the usability of the new feature before breaking old features.

That said, with the new user-specified versions you're no longer restricted to making dev releases. You can append additional dot separated components easily using the --version flag. If you don't want auto-incrementing, just don't use --final or --rebase.

karlkfi commented 10 years ago

After checking the code, it looks like my assumption about the default behavior (without --version, --rebase, or --final) was wrong. It looks like in that case the dev version gets auto-incremented, rather than just using the one in the manifest. So it looks like you have to use --version if you want to skip auto-incrementing.

karlkfi commented 10 years ago

For the record, specifying the version on the command line (via --version) is currently limited to final releases. @goehmen Would it make sense to loosen this restriction to allow specifying new custom pre-release or post-release versions via cli?

maxbrunsfeld commented 10 years ago

Hi @duglin,

As to your question regarding the differences between dev and final releases, a final release currently requires access to some blobstore and a dev release can be rebased by the director such that the version of a release will be changed at upload to not conflict with previously uploaded releases.

To the question about who should make final releases, we were hoping that the epic would encompass enough possible workflows to enable people other than Pivotal to create final releases such that you at IBM could take cf-release v100 a final release with an arbitrary version. If when checking out the original cf-release from GitHub you should be able to change your blobstore configuration and create your own final releases with IBM bits. You may want to lobby @goehmen for the relaxation of the blobstore requirement when creating final releases by providing a --without-blobstore option or similar.

Regarding the auto-incrementation of the release version, you may have a point that BOSH is trying to be 'too smart' in order to add a little convenience, and perhaps it may make sense to deprecate the creation of final releases without specifying a version explicitly but given the magnitude of such a change it would certainly require a long sunset window if @goehmen, was interested in the feature.

With the introduction of user-specified semantic versioning in BOSH, we have come a long way in removing the specialness of final releases, such that dev is essentially now a builtin post release version specifier. We would welcome any additional input you have on the issue as additional stories are completed. Are there any specific additional features beyond the stories which currently exist that you require in order to close this issue?

Thanks, CF Community Pair Caleb and Max (@calebamiles, @maxbrunsfeld)

duglin commented 10 years ago

Well, let's go back to my original scenario, I pick up cf-release v172, make some modifications to the code in some of its submodules and then want to build a "ibm-v172.1" tarball of cf-release. I'm doing this using bosh-lite, so maybe that's an issue but "bosh create release" 's help text doesn't seem to mention passing in a version name/number. So, what's the command line I put into my automation scripts to generate a tarball where in the tarball's release.MF file I have "ibm-v172.1" for the top-level "version" string and the phrase "ibm-v172.1" in the "version" string for the rebuilt jobs - e.g. hm9000 has a "version" string of "15-ibm-v172.1"? It's been a while since I wrote my script, but I think I needed to modify the version string of changed submodules otherwise BOSH would complain about a mismatch.

karlkfi commented 10 years ago

@duglin With the new semi-semantic version support (https://www.pivotaltracker.com/story/show/62085266) I'd recommend taking advantage of the new the new pre-release version support, going from a pivotal release of "172" to an IBM release of "172.1-ibm.1". Then for dev releases it would add "172.1-ibm.1+dev.1". This would retain logical sorting with pivotal releases: "172" < "172.1-ibm.1" < "172.1-ibm.1+dev.1" < "172+dev.1" < "172.1" < "172.1+dev.1" < "173"

If you stuck with "ibm-v172.1" it would parse it thinking "ibm" was the primary segment and "v172.1" was the pre-release segment, and the "v172" component would sort alphanumerically. It should work otherwise, but sorting would be off (which would affect bosh cleanup or version: latest).

duglin commented 10 years ago

I'm not sure I really care too much about the parsing side of things since each new build is totally independent in my case - eg. brand new cf-release dir, no history in the dev_releases dir. The exception to that is the blobstore (I think) since, as I mentioned I needed to tweak the version string of the changed submodules to avoid a bosh conflict error message.

So, in the end, I'm not clear on how to get what I'm looking for. What is the bosh command line I would issue to build what I described above? (tarball with the correct ibm-.... release name) Or is it not possible?

Also, my "bosh create release --help" doesn't seem to mention a "--version" flag, I'm on BOSH 1.2200.0 via bosh-lite, do I need to wait for bosh-lite to catch-up?

karlkfi commented 10 years ago

The new user-defined version support is not yet released. There are several related stories in progress. When it is done and release, there will be a new bosh cli gem and new bosh stemcells (with the new director).

That said, bosh-lite uses its own custom stemcell with the old director. So bosh-lite will also have to be updated in order to use new-format versions. We just made a story to update it: https://www.pivotaltracker.com/story/show/72572098

Once user-defined versions are supported, if your custom release version doesn't sort well, you wont be able to use "latest" release, bosh cleanup may delete the wrong versions", and auto-incrementing in create release and upload release --rebase may not work, but most other things may still work for you, especially if you create release with a specified manifest or final version.

As it turns out, the user-defined version release also adds the "--version" flag I've been mentioning. Sorry for the confusion.

karlkfi commented 10 years ago

@duglin, @sykesm mentioned today that you probably meant that you wanted to use "ibm" as the release name, not in the release version as I had mistakenly inferred.

We discussed this morning that specifying a name would actually resolve most of the issue, and that custom versioning is somewhat tangential to your issues. It sounds like custom versioning might be usable to work around the issue, but custom naming would be more useful, and having both would be ideal.

Is that accurate?

duglin commented 10 years ago

and the release.MF in cf-release.tgz has: name: cf version: ibm-v172.1

which I think means I want to tweak the version string not the release name, right? Its important to me to keep the original release names (e.g. "cf") so that people who see our setup can easily map it to other CF installs. It also makes it easier if I pick-up a vanilla cf-release.tgz because the release name will still be "cf" and its less changes in my BOSH manifest if I want to easily move between CF's 172 and the ibm-v172.1. To me, that's all the same "release" just different versions of it.

karlkfi commented 10 years ago

Interesting... So you're already creating custom versions by modifying the manifest before creating a release? If so, then yes, that's the version string. It's just an unexpected usage.

Is there a technical or business reason why you don't want to name it something like "cf-ibm", other than having to modify another line in the release manifest?

duglin commented 10 years ago

Just so you have the complete picture, here's the snippet of bash we use to fix-up our release tgz,after we create the release, to make sure it has the version string we want: gzip -f -d $project_name.tgz tar -xvf $projectname.tar ./release.MF sed -i "s/.[0-9]-dev/-$RELEASEVERSION/g" release.MF sed -i "s/^name:./name: $release_name/" release.MF sed -i "s/^version:.*/version: $RELEASE_VERSION/" release.MF tar -rf $project_name.tar ./release.MF gzip $project_name.tar rm release.MF mv $project_name.tar.gz $project_name.tgz

where $project_name is "cf-release", $release_name="cf", $RELEASE_VERSION is "ibm-v172.1".

We would have done this before we created the release tgz but it wasn't clear, at that time, how to stop "bosh create release" from doing things like using "-dev" in the version string. So, it was just easier to fix it up afterwards.

As for not wanting to name it "cf-ibm".... to be honest, at the time I never even considered it. Probably because the help text of "bosh create release" never mentioned being able to specify a new name, but more because I don't consider it to be a different release, its a different version of the same release. The same way a dev build is just a new version of the same release, and bosh doesn't force those to be named "cf-dev".

But I still go back to the same question... why does bosh care what I call it? What is the reason for preventing someone from naming a release whatever they want?

karlkfi commented 10 years ago

I think BOSH already allows you to supply a manifest as a CLI parameter to bosh create release, in which case the name and version from the manifest are used and not modified or incremented, allowing you to specify them beforehand. Have you tried that?

I can't speak to "why", being new to the team, other than to guess that it doesn't yet allow you to specify them on the CLI itself, because it wasn't a known required use case, not that it was deliberately disallowed. The existing simple versioning strikes me as a product of implementing the simplist approach to solve the known requirements, which is even now our general approach to new features.

I do think it's probably reasonable tho to add a --name parameter, but I delegate those product decisions and prioretization to our @goehmen. A PR is probably the best way to go, but the code around versioning is in flux ATM. It might be better to hold off until the new versioning changes are complete before attempting it.

duglin commented 10 years ago

Nope, didn't try that, is it an undocumented CLI option?

bosh create release --help Usage: bosh [options] --force bypass git dirty state check --final create final release --with-tarball create release tarball --dry-run stop before writing release manifest

Using BOSH 1.2334.0 via bosh-lite.

With such an option, is there a generic algorithm I could use to generate a modified input manifest? From my limited knowledge of bosh it seems like there is no requirement for a manifest file to be present prior to the "bosh create release", so I don't always have something to copy and modify with a new version string. I do see one for cf-release but for other releases that we build I don't see one. And I'm not sure it makes sense to require an entire manifest file just to specify a version string. Also, isn't this usually a generated file rather than a hand-crafted one?

For example, we have an admin-ui-release that gets built and it ends up initially with a version string of 0.1-dev, I assume because its the first bosh build and there's no existing manifest file to tell bosh otherwise. And this works ok for us because we then blindly modify the release.MF in the resulting tgz to have an IBM-specific version string.

karlkfi commented 10 years ago

Looks like you've discovered an interesting discrepancy in how the help info prints...

$ bosh version
BOSH 1.2579.0
$ bosh help create release
create release [<manifest_file>] [--force] [--final] [--with-tarball]
[--dry-run]
    Create release (assumes current directory to be a release repository)
    --force                       bypass git dirty state check
    --final                       create final release
    --with-tarball                create release tarball
    --dry-run                     stop before writing release manifest
$ bosh create release --help
Usage: bosh [options]
        --force                      bypass git dirty state check
        --final                      create final release
        --with-tarball               create release tarball
        --dry-run                    stop before writing release manifest

TBH, I don't actually know why there's an option to specify a manifest. The only useful scenario I can think of would be to re-create a release after tweaking it manually, or for reproducing bug behavior given specific input at a later stage. I was just thinking it might work for your needs if you already have a manifest.

karlkfi commented 10 years ago

I created a bug for the help discrepancy: https://www.pivotaltracker.com/story/show/72714220

duglin commented 10 years ago

that help text discrepancy is kind of funky :-) gotta wonder why there are two ways of doing (almost) the same thing.

karlkfi commented 10 years ago

@duglin The release versioning changes are live. That code should be calmed down enough now for you to work on a PR to add a --name feature to create release.

There is an outstanding bug, however, where renaming a release without other changes seems to mess up version auto-incrementing, due to how the release index stores by fingerprint and not by release name. But you'll probably be using --version for your case, with other changes, so it might not matter. https://www.pivotaltracker.com/story/show/72994716

bosh-ci-push-pull commented 10 years ago

I pulled that bug into the backlog. Thanks Karl.

Greg Oehmen Cloud Foundry Product Manager - Bosh Pivotal

On Thu, Jun 12, 2014 at 6:08 PM, Karl Isenberg notifications@github.com wrote:

@duglin https://github.com/duglin The release versioning changes are live. That code should be calmed down enough now for you to work on a PR to add a --name feature to create release.

There is an outstanding bug, however, where renaming a release without other changes seems to mess up version auto-incrementing, due to how the release index stores by fingerprint and not by release name. But you'll probably be using --version for your case, with other changes, so it might not matter. https://www.pivotaltracker.com/story/show/72994716

— Reply to this email directly or view it on GitHub https://github.com/cloudfoundry/bosh/issues/578#issuecomment-45966195.

You received this message because you are subscribed to the Google Groups "pivotal-bosh-eng" group. To unsubscribe from this group and stop receiving emails from it, send an email to pivotal-bosh-eng+unsubscribe@pivotallabs.com. To post to this group, send email to pivotal-bosh-eng@pivotallabs.com. To view this discussion on the web visit https://groups.google.com/a/pivotallabs.com/d/msgid/pivotal-bosh-eng/cloudfoundry/bosh/issues/578/45966195%40github.com https://groups.google.com/a/pivotallabs.com/d/msgid/pivotal-bosh-eng/cloudfoundry/bosh/issues/578/45966195%40github.com?utm_medium=email&utm_source=footer .

cppforlife commented 10 years ago

Support for --name got introduced in c4d047a. Next bosh release should have it.

duglin commented 10 years ago

great! thanks! How does bosh-lite pickup features like this?

drnic commented 10 years ago

bosh-lite uses the same bosh repo but it's on a slower release cycle; and mcf (which trycf.starkandwayne.com uses) is on a slower cycle that that (months old)

@ccpforlife we'd be happy to help with bosh-lite/mcf releases if you'd like

On Tue, Aug 5, 2014 at 6:07 AM, Doug Davis notifications@github.com wrote:

great! thanks!

How does bosh-lite pickup features like this?

Reply to this email directly or view it on GitHub: https://github.com/cloudfoundry/bosh/issues/578#issuecomment-51194335

calebamiles commented 10 years ago

Doug,

Unless I'm mistaken, all work required to support named releases was done in the CLI so you should be able to do a 'gem update bosh_cli' after the next release to pick up the changes. Please let the team know if you encounter any issues using the new CLI with your BOSH Lite environment.

Caleb On Aug 5, 2014 7:03 AM, "Dr Nic Williams" notifications@github.com wrote:

bosh-lite uses the same bosh repo but it's on a slower release cycle; and mcf (which trycf.starkandwayne.com uses) is on a slower cycle that that (months old)

@ccpforlife we'd be happy to help with bosh-lite/mcf releases if you'd like

On Tue, Aug 5, 2014 at 6:07 AM, Doug Davis notifications@github.com wrote:

great! thanks!

How does bosh-lite pickup features like this?

Reply to this email directly or view it on GitHub: https://github.com/cloudfoundry/bosh/issues/578#issuecomment-51194335

— Reply to this email directly or view it on GitHub https://github.com/cloudfoundry/bosh/issues/578#issuecomment-51201327.

cloudfoundry / bosh

Add support for specifying a name during bosh create release #578

How does bosh-lite pickup features like this?

How does bosh-lite pickup features like this?