Convert .deb build from Ansible to Docker

[heartsucker]

Feature request

Description

We use molecule + ansible to build our debian packages. This is a bit annoying because there is little caching that is saved between builds. This means the dev + build + deploy + test cycle is very slow (10 minutes), which means changing a single line in a postinst script is tedious to test.

We should move our build logic into Dockerfiles so we can take advantage of caching. We would still use molecule as the wrapper that handles the builds, but the bulk of the logic would be in the Dockerfile itself. If the last steps are copying in the app code and DEBIAN dir, subsequent builds will be only seconds long.

Tagging @msheiny since he probably has opinions on this. :D

User Stories

As a developer testing deployment / builds for the Debian packages, I want better caching so I can spend less time waiting for builds to finish.

[msheiny]

:+1: I'm not a fan of the current ansible scripts however I do think ansible can continue to call docker and we should continue to use molecule to wrap it so we can continue to utilize our testinfra tests. Basically I want to keep the testinfra tests in the flow because i dont want to lose test coverage in this migration.

[heartsucker]

Ok, yeah so molecule -> ansible -> docker would be how we do the build. Not yanking ansible entirely, but things like "install build dependencies" and so on would happen in the container so we cache them between builds.

[conorsch]

:+1: on switching to Dockerfiles for the package build logic. The lack of caching there is maddening. Optimistically we'd get parallelization (#2939) for free, as well.

want to keep the testinfra tests in the flow because i dont want to lose test coverage in this migration.

Agreed, but wiring up the Docker/Molecule integration could be tricky. As a fallback, we could we run the testinfra tests against the deb packages on the host machine, presumably placed there by the build containers via volume mount.

[msheiny]

Agreed, but wiring up the Docker/Molecule integration could be tricky

Yeah -- i mean we do it all the time but I agree the caching logic sucks with ansible + docker. You have to do all these stupid work-arounds to get it to intelligently rebuild.

[conorsch]

What if we just expand the Dockerfile template to include the required packages? We'd probably need more than one template, particularly for the app-code deb, but for the others they should be pretty minimal. We already have the install_files/ansible-base/roles/build-generic-pkg/files/build_generic_package.sh script that's perfect for containers.