Closed samrose closed 1 month ago
@darora just wanted to follow up that in this new PR testinfra ami tests are now passing for the nix ami build
https://github.com/supabase/postgres/actions/runs/9672036422/job/26683662377?pr=1012#step:10:95
This resolves https://github.com/supabase/postgres/pull/953#discussion_r1650347893
I have moved the docker work to a new PR that should be coming up tomorrow (and will deprecate the old)
In my view we still need some automatic deduplication of config files to be sure they don't have differences and will not have in the future. E.g ansible-nix/files/kong_config/kong.service.j2 and postgres/ansible/files/kong_config/kong.service.j2 look like copies and many other files like that. If we have files derived from other ones, I think it's better to have one "master" one and the other to be derived in an automatic way. Otherwise, we could get easily confused in the changes between them. And also this make PR somewhat bulky and hard to review. Maybe a script that derives config files from main Supabase repo is a suitable solution for this. Or symlinks to existing files.
@pashkinelfe that's reasonable to me, thanks.
The diff itself looks fine, though someone will need to test an actual upgrade to see if there's any other hurdles that are not immediately obvious. Agreed with Pavel if it's doable in a straightforward manner. Otherwise, I guess we'll have to review a diff of the two dirs locally as a one-time exercise.
This PR will need minor follow up prior to approval/merge for github actions that are dedicated to specifically merging to develop. It supersedes pr #953
Documentation of changes in https://github.com/supabase/postgres/pull/1012
Conventional AMI approach
The existing/conventional AMI build approach installs postgres from the
postgresql-common
ubuntu/debian package at the time of the AMI build. In addition, it builds extensions, and wrappers from source at the point of AMI build, and installs them as ‘.deb’ packages.Nix packaged postgresql bundle approach
In the nix approach, we use the postgresql provided by nixpkgs (currently pinned at version 15.6 vi a76c4553d7e741e17f289224eda135423de0491d commit of
nixpkgs-unstable
branch locked via https://github.com/supabase/postgres/blob/sam/2-stage-ami-nix/flake.lock#L114 )Nixpkgs sources from https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/generic.nix#L52 ← this URL
the nixpkgs package applies the following patches for
aarch64-linux
pg 15.6https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/disable-resolve_symlinks.patch
https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/less-is-more.patch
https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/hardcode-pgxs-path.patch
https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/specify_pkglibdir_at_runtime.patch
https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/findstring.patch
https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/locale-binary-path.patch
https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/socketdir-in-run-13.patch
When a PR is submitted to the supabase/postgres repo updating any of the nix packages maintained there, a build of the entire bundle is triggered on supported systems (
x86_64-linux
andaarch64-linux
as of this writing). When this the nix ci workflow is initiated, nix is able to source from our binary cache (currently located in a publicly readable aws s3 bucket at https://nix-postgres-artifacts.s3.amazonaws.com ) and will check for any component dependency which has an exact match and has already successfully built. Nix will source that built version from the cache, and only build the items that have changed. If nix cannot build a changed item, the build will fail. If the build succeeds, nix will perform flake “checks” (scripted tests with dependencies managed by nix). An example of the “check” is seen hereOur CI implmentation of nix has only 2 trusted public keys and 2 specified nix caches (ours and the upstream nixpkgs community cache https://github.com/supabase/postgres/blob/sam/2-stage-ami-nix/docker/nix/Dockerfile#L5 and on the AMI at https://github.com/supabase/postgres/blob/sam/2-stage-ami-nix/scripts/nix-provision.sh#L20 )
https://github.com/supabase/postgres/actions/runs/9468138054/job/26083806922?pr=953#step:6:813 this starts the database, and enables several extensions post-build. If this test fails, the build will also fail. If the nix build and check succeeds, the build will upload the artifacts to the nix cache for re-use prior to stopping this workflow.
In the debian/ubuntu
postgresql-common
package, the “postgres” user is created, and postgres is installed to locations that are conventional for debian/ubuntu. In the nix approach, we explicitly create the “postgres” linux user, and then we use thenix profile install
method to install the nix-built binaries for postgres, into thenix profile
for the “postgres” user (located a/home/postgres/.nixprofile
on the ami machine). We then alias the installed file locations to the conventional debian/ubuntu locations for postgres installation.nix profile
command will give us an imperative way to install, uninstall, and upgrade packages that we build with nix going forward, allowing us to integrate our nix-built packages with debian/ubuntu distributions.2 Stage AMI approach
The Ansible and Packer code has been forked in parallel in the same repo, so that both the nix-built approach, and the existing ubuntu/debian package approach can be supported in paralell. This will allow continued production rollouts under the old method, while also allowing targeted rollouts with the nix build AMI.
The existing build lives under the same
ansible
folder, and the companion packer hcl files have been retained. The parallel nix AMI build has parallel packer files withnix
inserted into the name, and a new folderansible-nix
. Both of these builds use the same command line command recipe to initiate them.Description of 2 stage approach
The previous packer/ansible build used the https://developer.hashicorp.com/packer/integrations/hashicorp/amazon/latest/components/builder/ebssurrogate exclusively. The new nix-based retains the ebssurrogate approach to build and configure everything except for the postgres bundle.
The nature of nix builds is that they are already “sandboxed” and isolated at build time, and the results are store in a read only directory called the “nix store”. Nix has never had the need to support building in
chroot
as ebssurrogate packer build does, and so running nix inchroot
has never been supported for these reasons. Therefore, a second stage of the AMI build was introduced, that securely sources the private “stage1” AMI built by the stage 1 ebssurrogate approach, and then installs the nix built suapbase postgres/extensions/wrappers bundle from binary cache using the conventional github.com/hashicorp/amazon packer plugin, and limited to installing, configuring and testing postgres from either files uploaded in the first stage, or sourced from nix cache (other than stage 2 ansible playbook and unit test files). The workflow that performs these 2 stages is located here https://github.com/supabase/postgres/blob/sam/2-stage-ami-nix/.github/workflows/ami-release-nix.yml As more Supabase projects are packaged in nix, they will be moved into this 2nd stage for installation and configuration. In the 2nd stage we run migration and unit tests, and linux user/group assignment checks with a temporarily installed copy of osquery https://github.com/supabase/postgres/blob/sam/2-stage-ami-nix/ansible-nix/tasks/stage2/playbook.yml#L71 and https://github.com/supabase/postgres/blob/sam/2-stage-ami-nix/ansible-nix/files/permission_check.pyThe 2nd stage also creates path aliases to the nix-installed binaries so that files and configurations are still where they are expected to be as much as possible. This allows post-AMI-build init scripts like https://github.com/supabase/infrastructure/blob/develop/init-scripts/project/00-init.sh to continue to succeed in running.
We are maintaining documentation on how to work with the nix portion of supabase/postgres at https://github.com/supabase/postgres/tree/sam/2-stage-ami-nix/nix/docs and will continue to expand that as much as possible.
Current Progress on adoption in https://github.com/supabase/postgres
There is an umbrella draft PR at https://github.com/supabase/postgres/pull/953 which includes the building of an aarch64-linux AMI
Docker image PR https://github.com/supabase/postgres/pull/986
Docker AIO Image PR https://github.com/supabase/postgres/pull/987
The Ansible and Packer code has been forked in parallel in the same repo, so that both the nix-built approach, and the existing ubuntu/debian package approach can be supported in paralell. This will allow continued production rollouts under the old method, while also allowing targeted rollouts with the nix build AMI.