Stemcell builder improvements

[allomov]

Currently it is hard to build stemcells, especially if you want to build them to a new clouds.

There are several reasons for it:

• you have to patch bosh, I had to patch bosh-release, bosh-stemcell, bosh-dev projects except of stemcell_builder.
• it takes a lot of time to check your changes had necessary effects
• you can’t build “light” stemcells for new infrastructures easily using current BOSH stemcell builder.

I suppose that live would be much more better after BOSH will go from compiling MicroBOSH into stemcell, but still there are some possible ways to improve it.

Here are some suggestions to improve stemcell builder:

• Run stemcell builder locally. I had customers that didn’t wanted to use any external infrastructure (like AWS) to build stemcell, this meant they wanted to run them into their IaaS. Vagrant box didn’t played well in such situations because of absence of nested virtualization.
• Build “light” stemcells for external clouds. Some clouds doesn’t have “custom images” as a feature. And the only one option there is to build instance templates from existing VMs. For this kind of solutions it is necessary to run stemcell builder on the cloud instence and build stemcell into attached disks or a system disk.
• Rollback of state of file system after failed stages. Currently you spend a lot of time rebuilding stemcell after small changes that leaded to errors. I suggest to use something like aufs to keep state of file system to rollback after stage failed. This will speed up development process.
• Declarative way to define stages. That's ok to have such DSL as BOSH currently has, but it would be much more easier for people not to mess with ruby code in stage_collection.rb file, but to pass some .yml manifest to the builder.

Also I have a strange feeling that it would be better to rewrite stemcell builder in go lang :)

What do you think ?

[cppforlife]

I suppose that live would be much more better after BOSH will go from compiling MicroBOSH into stemcell, but still there are some possible ways to improve it.

Yes. We already started to benefit from it, when it's necessary to make CPI changes. The only time it's necessary to build a stemcell now is if you want to change the agent or make changes to the OS.

Run stemcell builder locally

Was docker available in such environment?

Rollback of state of file system after failed stages

Have you see resume_from mentioned in https://github.com/cloudfoundry/bosh/tree/master/bosh-stemcell#when-things-go-sideways?

Declarative way to define stages

Agreed. I do want to preserve certain level of enforcement that all Ubuntu's are the same, regardless which cloud it's for.

[allomov]

Yes. We already started to benefit from it, when it's necessary to make CPI changes. The only time it's necessary to build a stemcell now is if you want to change the agent or make changes to the OS.

Does it mean that having MicroBOSH release in a stemcell is not mandatory at the moment? I think of making PR to support building stemells for Power platform, still it misses MicroBOSH release build stage. Do you think it's okey to make a PR without this stage?

Was docker available in such environment?

Yes, basically Docker was available. In the same time I had a case when I couldn't make any custom images for customers IaaS, to solve this issue I needed to create VM which was bootstrapped by stemcell builder and this VM was cloned during create_vm stage. I don't say that this scenario is a good way to go, but at this moment it was the only way.

Have you see resume_from

Thank you for referencing to resume_from, still I started to work with stemcell-builder before this feature was added and I made some rake tasks to run distinct stages.

[allomov]

I have one more reason to quit using VirtualBox / AWS for stemcell building: stemcell-buider doesn't allow to build a stemcell for a platform that is not supported by VirtualBox (in my case it is Power). To maintain such solution you need to patch stemcell-builder and to implement solution to bootstrap environment for it (in case of Power I needed to write ansible playbooks).

[cppforlife]

@allomov we now have experimental stemcell building pipeline: https://main.bosh-ci.cf-app.com/pipelines/stemcell-new-dev-temp that builds stemcells in docker container. you should find pipeline config in ci/ folder of the bosh repo.

[allomov]

Thank you for this message. I guess this we will have an opportunity to check it out very soon.

[dpb587-pivotal]

Closing due to age - the stemcell builder has undergone significant changes since this discussion. Feel free to re-evaluate and open a new issue in cloudfoundry/bosh-linux-stemcell-builder if you want to discuss specific suggestions now.