Building with ko instead of docker breaks the released image when an existing database is used

[ekeih]

Describe the bug

After upgrading to v3.22.1 the backend crashes with error during DB migration: attempt to write a readonly database. This happens because the backend is now running as user nonroot with uid and gid 65532 (or at least that is what I think based on the documentation, see below).

To Reproduce Steps to reproduce the behavior:

1. Update the backend to the v3.22.1 docker image.

Expected behavior The backend should not crash and it should be able to write to existing sqlite databases.

Additional context

• With https://github.com/envelope-zero/backend/commit/cb5ad545cd93162f8e7e3e9fd05a80fcd962939e the build process was changed from docker to ko. The existing Docker image is not used by ko at all, it can be deleted when using ko.
• ko uses cgr.dev/chainguard/static as base image for all builds by default (according to https://ko.build/configuration/#overriding-base-images).
• According to https://github.com/chainguard-images/images/tree/main/images/static this base image has a single user nonroot with uid 65532, belonging to gid 65532.
• Before the image was build with docker based on https://github.com/envelope-zero/backend/blob/a81350dce9b8b6df32ef51f8a1a65f6ab73b63f4/Dockerfile.goreleaser, which simply used the scratch image with the default user/group root.
• The existing sqlite database belongs to root (uid 1), so the new backend version running as 65532 does not have write permissions to the existing sqlite file.
• I suspect that this does not break for new users, because their initial database could be written to a new volume. (The volume permissions might depend a bit on how the backend is deployed.)

Switching to scratch with ko is not possible

• My initial idea was to create PR to override the base image of ko to the scratch image.
• Based on https://github.com/ko-build/ko/issues/207 and https://github.com/ko-build/ko/pull/1038 I suspect that this is currently not possible.

Proposal

• Running the backend as a non-root user is actually a good thing.
• We could keep the new base image and "fail forward".
• This would be (or already is) a breaking change because existing users would need to change the file system permissions of their sqlite database before upgrading.
• The cgr.dev/chainguard/static image only has a latest tag, so future changes in the base image could cause issues in the future. Though, the image should be fairly minimal overall. An alternative could be to use another base image or create our own.
• Depending on how long a decision/fix takes, we should consider reverting back to docker in the meantime.

[morremeyer]

Thanks for researching the issue this thoroughly!

With all this information, I agree with your proposal of failing forward. This decision is also made easier by the fact that the release of v4.0.0 is just days away, if at all.

However, I would also like to revert back to docker for now so that we have a clean upgrade path (upgrading the major will entail upgrading to the latest patch version of the major version before from v4.0.0 on).

Two questions:

• What do you mean by The existing Docker image is not used by ko at all, it can be deleted when using ko., can we delete the Dockerfile.goreleaser with this?
• Since cgr.dev/chainguard/static only has a latest tag, did you see a way to pin the digest of the image somewhere during your research?

[ekeih]

However, I would also like to revert back to docker for now so that we have a clean upgrade path (upgrading the major will entail upgrading to the latest patch version of the major version before from v4.0.0 on).

Sounds like a good idea 👍

What do you mean by The existing Docker image is not used by ko at all, it can be deleted when using ko., can we delete the Dockerfile.goreleaser with this?

Ah sorry, I meant to write Dockerfile. Yes, we can delete Dockerfile.goreleaser when we switch to ko with v4. https://ko.build/advanced/migrating-from-dockerfile/ says You're done. You can delete your Dockerfile and uninstall docker. at the bottom.

Since cgr.dev/chainguard/static only has a latest tag, did you see a way to pin the digest of the image somewhere during your research?

I didn't see any documentation that this is possible. I also didn't see any saying it is not. But my gut feeling is that it is not possible.

[ekeih]

ℹ️ I stopped my backend instance, ran chown 65532:65532 on the sqlite file and its parent directory, started the new backend version (v3.22.1) and everything worked.

[morremeyer]

I added the change back with #915, check the upgrade docs for v4.0.0.

Thanks again for the research!