AMI kickstarts

[jantman]

I'm trying to build a VirtualBox/Vagrant image as close as practical (in terms of installed packages) to the official AMIs. It would be greatly appreciated if the kickstart files used for the official AMIs could be added.

[jantman]

Any chance that this can be looked into? @kbsingh ?

[kbsingh]

hey @jantman - we dont have the centos-7 ami's up as yet, am working to get those done this week ( well, over to the aws people so we can get them listed ). As soon as that happens, I'll sync the kickstarts over.

Would also be great to have vagrant images, with various backends.

In the mean time, I can try and get the work-in-progress kickstarts up, so you can get started on the things you need to get done.

[jantman]

@kbsingh I was actually looking for cent6... haven't done much with 7 yet.

I could probably provide some assistance with virtualbox vagrant images, if it's needed.

[mvermaes]

Hi, are these kickstarts available anywhere for the CentOS 7 AMI?

[blalor]

This would be very useful.

[elyscape]

@kbsingh Any updates on this? It would be nice to be able to build custom AMIs. Additionally, having these publicly available might help get the official AMIs to be updated more often. Thanks for your help!

[kbsingh]

The Vagrant images should be identical to the AMIs already - which should be identical to the GenericCloud images posted on cloud.centos.org/centos/7/images/ ; the only factor that changes is the backing file format, and in the Vagrant images, there is no cloud-init ( and therefore its deps ).

does that help resolve the issue ?

[kbsingh]

@elyscape part of the challenge in the official images being updated is down to the process we need to use for the marketplace listings. Its something I want to work on better, and also ensure we get better region cover in the short term.

[blalor]

I don't understand why it's so difficult to just share the exact scripts and config required to build the official AMIs.

[kbsingh]

@blalor eventually, i want to get to building these in cbs.centos.org - but were not there yet, and the infra needed ( its not just scripts and configs ) is cumbersome to untangle from the other bits around it. building them in cbs.centos.org should reduce the overall metadata to just the disk image format conversions and a kickstart file.

[alanivey]

I had brought this up on centos-devel and was able to import a GenericCloud image into EC2 successfully (and, I'm happy to share how I did so in another forum if anyone is curious).

As for the exact scripts used to create the AMIs; this would be very helpful for transparency in reaching compliance in many programs. I understand the desire to release them in the best way possible, this issue has been open for over a year, and I've seen about 6 or 7 threads on the CentOS mailing lists asking for this insight without resolution.

@kbsingh Until things are tidied up to meet your objectives, could you perhaps drop the current scripts into a private Gist and post the link here in this issue? Personally, I'm held up on a project where I need additional insight for compliance purposes.

[cryptickp]

@alanthing Could you please share how you imported GenericCloud image into EC2, I'm facing same difficulty.

[alanivey]

@cryptickp: Posted to https://gist.github.com/alanivey/cd199758a759267c98fe

You'll need to determine the S3 bucket you want to use, create an IAM role and reference the bucket, create an IAM user (with API keys) with inline or attached policies referencing the bucket (if not using a PowerUser or Administrator, etc), change the values in the script for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and S3_BUCKET, and optionally change the AMI Name or Description.

You'll notice that after the ImportImage completes, I Copy the AMI and then delete the Imported one. That's because it's not possible to set the Name or Description with an Import- only with a Copy. Also, the imported AMI has the Platform as "Other", while the copied AMI has the Platform as "Cent OS". I guess AWS determines the value based on the name or the description from a whitelist of acceptable Platforms.

Let me know if you have any questions.

@kbsingh: I'm happy to lend my support in cleaning up your other scripts; the gist should provide some insight into how I could help.

[chasebolt]

can we please get the kickstart file for creating the centos 7 ami? i really need this for consistency / compliance across multiple infrastructures.

[earsdown]

Can we get an update on this please. A lack of transparency in such a critical area can quickly erode trust and confidence in CentOS itself. The last thing anyone wants is to call into question the trustworthiness of the official CentOS images. Please release these kickstart files as urgently as possible.

[alanivey]

@earsdown I ended up reverse-engineering the AMI and built my own AMI from scratch (mounting another EBS volume, using yum and rpm with a chroot on the new volume, snapshot from that volume). I did extensive comparisons between an instance with the official AMI and my own until I was confident they are as close to parity as reasonably possible. Definitely less than ideal.

My notes are not ready for sharing, but perhaps these resources that I used can help you:

• https://github.com/ferricoxide/AMIgen7
• https://github.com/eschwim/centos7-ami-builder

[earsdown]

@alanthing thanks for that. Because it's a relatively infrequent process, I've been playing around with building a local VM from the CentOS DVD ISO then using the VM import capability. This also seems to be working, and I didn't have to mess around with kickstart. Even so, if we're going to go around suggesting companies use the official images, it doesn't look good if we can't review and reproduce the build ourselves.

[aspring]

On the Centos 7 AMI side, we ended up reverse engineering as best we could a kickstart file, then made our own internal tweaks. That kickstart gets passed into virt-install, and the resulting disk image(s) are passed into a slightly modified version of: https://github.com/blalor/ami-creator/blob/master/utils/create-ami.sh to create an HVM AMI.

All of this gets run inside a vagrant box -- so creating a new AMI is as simple as updating the necessary kickstart pieces, and doing a vagrant up.

This gives us end to end control of AMI generation.

[blalor]

@aspring glad that's useful to you! What'd you need to modify?

[aspring]

Superficially I split it into PV and HVM scripts to make understanding and debugging easier, and added a small getopts based CLI.

From a guts side, I left PV the same. For HVM I build the disk images with virt-install, so the build process was pulled from the script and functionality added get the disk images into AWS with import-volume, and I added multiple disk support.

Ive dropped a gist of the HVM script here: https://gist.github.com/aspring/720a0df5680a7fb8afc599e739f822c1

[kbsingh]

You guys have looked at this right : https://github.com/CentOS/sig-cloud-instance-build/blob/master/cloudimg/CentOS-7-x86_64-GenericCloud-201606-r1.ks

[alanivey]

@kbsingh -- thank you, this is very helpful! I do not believe this was available until a few months ago so I didn't even know it was here.

In a previous exchange, you had said that some small changes are made before this GenericCloud KS-ed image is pushed up to an AMI. Are you able to comment on what those changes are?

[zephyia]

Hi guys, Is there any newer links available yet that have the kickstart (and any other potential steps) for building an AMI as close as possible to official centos AMIs?

[kbsingh]

@zephyia are you not able to just import the generic-cloud qcow2 images ? that would give you the same result