Need an improved chroot-builder

ferricoxide commented 6 months ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Please search the existing issues for relevant feature requests, and use the reaction feature (https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to add upvotes to pre-existing requests.

Description

Need the ability to have a chroot-build method that supports a workflow that looks like:

Launch build-EC2 with root and build-target EBS attached
Execute a build to the secondary EBS
Power off the build-EC2
Detach boot and build-target EBS from build-EC2
Re-attach build-target EBS as build-EC2's (new) boot-EBS
Register stopped build-EC2 as image
Terminate build-EC2
Clean up original boot-EBS and any other resources created for build

Use Case(s)

In order to inherit AMI/EC2 attributes from the bootstrap AMI – primarily concerned with the billingProduct – it's necessary to have this kind of EBS-switching workflow. The current amazon builder options don't afford this.

Previously, we used the amazon-ebs builder combined with a "pivot-root" preparation-script and a disk-releaser stage in our workflow. This allowed simply re-writing the contents of the boot-disk.

Unfortunately, it appears that there's a bug that got introduced in the Linux 5.x kernel (and that also seems to exist forward into at least the 6.1 kernel) that prevents the kernel from recognizing live/on-the-fly disk-layout changes made to the boot-disk. While issues have been opened with Red Hat (impacts their 5.14-based RHEL 9.x releases) and AWS (impacts their 6.1-based Amazon 2023 images) and each vendor has reproduced the problem, no determination of if or when a fix has been given by either vendor.

Ultimately, using the amazon-ebs builder the way we did was kind of hackish, but, "it worked". Prior to the recently discovered issues when attempting to extend our automation to RHEL 9.x and Amazon 2023 AMIs, it never necessitated looking at other Amazon Packer-builders. The current amazon-chroot looks close to what we need, but seems to lack the ability do anything but steps 1 & 2 (as described above). Steps 3+ are the key to producing images that inherit attributes from the bootstrap AMI. If we had an improved chroot-style build, we'd probably move all of our AMI-build recipies to that builder-option.

Potential configuration

Potential References

Inheriting billingProduct attribute
Red Hat Issue ID RHEL-17421

I would link to the Amazon issue ID, however, it is not subscribable outside of the account it was opened from

ferricoxide commented 6 months ago

Cc'ing my project-teammates for any input they might have @lorengordon @eemperor

ferricoxide commented 6 months ago

Oop... should have opened this against hashicorp/packer-plugin-amazon: can someone transfer?

nywilken commented 6 months ago

Oop... should have opened this against hashicorp/packer-plugin-amazon: can someone transfer?

@ferricoxide thanks for detailed request. I'll transfer this over now.

It's unlikely that we will bel able to prioritize this request at this time but we would gladly help review a PR. Given your team's knowledge on the workflow would you or anyone from your team be open to working on this request?

I need to spend some cycles wrapping my head on the ask but if you can work on a PR that would be great. From your description it seems like this would be additive so I wouldn't see an issue merging, as long it doesn't break the existing chroot behavior.

lorengordon commented 6 months ago

It's unlikely that we will bel able to prioritize this request at this time but we would gladly help review a PR. Given your team's knowledge on the workflow would you or anyone from your team be open to working on this request?

Thanks @nywilken. I've already assigned the investigation to a teammate. Proposing a new builder, I think, maybe ebs-chroot due to combining things the two existing builders do. Otherwise, could maybe enhance one of the existing builders to accommodate the workflow using an argument? Any thoughts/preference on approach?

ferricoxide commented 6 months ago

Proposing a new builder, I think, maybe ebs-chroot

Given the (apparent) RHEL kernel-bug and yet-to-be-determined timetable for remediation from the vendor, this likely effects other CSPs' builders. Might be good, once a suitable Amazon-builder has been offered/merged and has established a coded-approach, to also have one for Azure and other CPSs? I mean, I'd love to be able to avoid the current pivot-root shim, altogether. Dunno if it's too early to open corresponding issues in the other CSPs' packer-plugin projects, though.

dwc0011 commented 6 months ago

@nywilken I am going to begin looking into and working this. I wanted to see if you had a preference on the solution that @lorengordon proposed before I get too deep.

Thanks @nywilken. I've already assigned the investigation to a teammate. Proposing a new builder, I think, maybe ebs-chroot due to combining things the two existing builders do. Otherwise, could maybe enhance one of the existing builders to accommodate the workflow using an argument? Any thoughts/preference on approach?

dwc0011 commented 4 months ago

@nywilken I have implemented a swap volumes solution for the root volumes in ebs-surrogate. The PR is linked into please take a look when you have time as I continue to look into other tests I can implement.

Thanks!

lorengordon commented 4 months ago

It's unlikely that we will bel able to prioritize this request at this time but we would gladly help review a PR. Given your team's knowledge on the workflow would you or anyone from your team be open to working on this request?

Thanks @nywilken. I've already assigned the investigation to a teammate. Proposing a new builder, I think, maybe ebs-chroot due to combining things the two existing builders do. Otherwise, could maybe enhance one of the existing builders to accommodate the workflow using an argument? Any thoughts/preference on approach?

After further investigation, we realized that the ebssurrogate builder is closest to the desired approach. It already launches an ec2 instance, and expects the user to create a bootable disk on a secondary attached ebs volume. All we did was add an argument image_method that accepts values of create or register. When set to register (the default), it behaves as before, using RegisterImage. When set to create, it detaches the root volume and attaches the secondary volume as the root, and then it uses CreateImage on the instance.

hashicorp / packer-plugin-amazon