Supporting customization of tags in the built-in images

[brownleej]

The operator does not currently support customizing the tags on the foundationdb and foundationdb-kubernetes-operator image. This can be a difficult limitation in some scenarios:

• Building one-off images for testing
• Building variant images, like a slim image and a full image

Allowing arbitrary tags will cause problems when doing upgrades, since the operator needs to carefully manage the image versions to upgrade the sidecar and the main container at the appropriate stages.

This came up in #342, and we addressed the problem by adding stronger validations. I think we should take another look at it and see if there's a cleaner way to support some customization of tags.

[brownleej]

We could look at this problem as being similar to what we did for the sidecarVersions field, where we have suffices that are appended to the image on the basis of the version. We could generalize this somewhat to support both the sidecar container and the main container, and support arbitrary suffices, or maybe even arbitrary tags entirely, that are still scoped to a specific version of FDB.

[brownleej]

Re-opening this to follow up on the discussion in #587.

[brownleej]

Here's what I've been thinking in terms of how we can make this more flexible. We can create a new field called imageVersions that allows customizing docker images on a per version basis. This will be contain two fields, main and sidecar. Each of these fields will be a list of ImageConfig objects, which will have the following fields:

• version: The version of FDB this configuration applies to. If this is blank, it applies to all FDB versions.
• baseImage: Overrides the base image, i.e. the part of the image before the tag.
• tag: Specifies an exact image tag.
• tagSuffix: Specifies a suffix to append to the tag after the main FDB version.

When we fill in the images for a built-in containers, we should apply all of the matching rules specified by the imageVersions settings, in the order in which they are listed. A rule will match if its version is blank or matches the FDB version the cluster is running.

This will allow us to support the following scenarios:

• Specifying a custom base image for all FDB versions, to support someone running a private repository that mirrors the public images.
• Specifying a custom tag for an unreleased FDB version, to support images that are built off of the main branch.
• Using the official images for released versions and then upgrading to a custom image for an unreleased version.
• Specifying custom sidecar build suffices based on the FDB version, replacing the sidecarVersions field we have today.

If the user specifies an image for the container, but no tag, we should construct the tag based on the rules above. If the user specifies an image with a tag, we should respect that image in full.

@johscheuer What do you think?