Push multiple tags for the same image

[fejta]

Do you have a recommended recipe for pushing both a :immutable_based_on_commit and :latest tags at the same time?

In the kubernetes/test-infra repo our images follow a v20170807-deadbeef type pattern based on the date and commit. However whenever we push these images we also update the :latest image.

Our non-production jobs tend to use the :latest image. After we validate our test/canary jobs work, we then update the production deployments use the v20170807-deadbeef tag for this image.

Any suggestions on how you would like us to solve this pattern? It would be nice if a single rule could push both tags for the same image.

Example of this dual push strategy: https://github.com/kubernetes/test-infra/blob/master/images/e2e-prow/Makefile#L28

[mattmoor]

Just to summarize our offline conversation, I think this is possible today.

A little while back, I created a Bazel macro in docker/contrib/with-tag.bzl, that supports tagging an image in addition to building it. At its heart, this macro is just:

def docker_build(name=None, tag=None, **kwargs):
    docker_build_(name=name + '-internal', **kwargs)
    docker_bundle(name=name, images={
      tag: ':' + name + '-internal'
    })

I also created a variant of docker_push that works with bundles, in particular it publishes all of the embedded tags (example).

I think what you want is a combination of a small modification of the former with the latter. The change you want to the former is:

def docker_build(name=None, tag=None, **kwargs):
    latest = tag.rsplit(':', 1)[0] + ':latest'
    docker_build_(name=name + '-internal', **kwargs)
    docker_bundle(name=name, images={
      tag: ':' + name + '-internal',
      # Also tag the repo with latest
      latest: ':' + name + '-internal'
    })

You should be able to create a .bzl file in the K8s repo for this which is a bit cleaner and hides the commitish logic from BUILD files by taking the repo instead of the tag:

def docker_build(name=None, repo=None, **kwargs):
    docker_build_(name=name + '-internal', **kwargs)
    docker_bundle(name=name, images={
      repo + ':' + commitish(): ':' + name + '-internal',
      # Also tag the repo with latest
      repo + ':latest": ':' + name + '-internal'
    })

Hopefully this helps, I'm going to close this, but feel free to reopen if you need any changes to this repo.

[joshburkart]

I think I'd like a user to be able to execute a container_push with an arbitrary set of tags to apply at the command line... Maybe also different registries as well... E.g. given a rule like

container_push(
    name = "push_bar",
    ...
)

I'd like to be able to do something like the following on the command line:

bazel run //foo:push_bar --tag=latest --tag=built_by_josh --tag=prod

@mattmoor I don't think the solution you mention above would work for this? Since it requires the number of tags to be known in code, rather than at execution time? I'm motivated by how I used to build MPMs inside Google, where I'm pretty sure I could set a bunch of labels at once...

[joshburkart]

And while I'm at it, I'd love to be able to specify the registry to push to using an alias -- like "prod", "staging", etc. -- at the command line (with the mapping of "prod" -> a particular GCP project ID stored in a .bzl file somewhere). Not sure if the Makefile substitution method mentioned in the README allows this kind of functionality?

[mattmoor]

I don't have a good solution for variable number of tags, but you could certainly use something like:

container_bundle(
    name = "foo",
    images = {
        "{STABLE_REPOSITORY}/image:latest": "//path/to:image",
        "{STABLE_REPOSITORY}/image:{BUILD_USER}": "//path/to:image",
        "{STABLE_REPOSITORY}/image:{STABLE_ENVIRONMENT}": "//path/to:image",
    },
)

In this example {BUILD_USER} is a built-in "status" variable, and {STABLE_REPOSITORY} and {STABLE_ENVIRONMENT} would be variables you could inject via --workspace_status_command=./print-workspace-status.sh (put this in .bazelrc in your repo's root).

The make variable stuff just gets annoying because we didn't open source vardef to default them, which means your build is broken without --define FOO=default, so I tend to prefer status vars with Bazel.

cc @ixdy who taught me everything I know about status vars.

[joshburkart]

Hmm ok thanks for the info @mattmoor!

Actually, since running a container_push rule presumably executes some arbitrary script that you control in this repo, why couldn't it accept arbitrary command-line args? Why the need for using intrinsic, seemingly obscure Bazel workspace status stuff? Am I missing something? E.g. couldn't container_push be configured to work like...

bazel run //foo:push_bar -- \  # The "--" passes subsequent args to the `container_push` script...
  --tag=latest \
  --tag=built_by_josh \
  --tag=prod \
  --registry=http://gcr.io/something-or-other

This seems like it could be quite flexible/convenient?

[mattmoor]

Where it's tricky is which tags apply to which images in the bundle? Right now it's pretty dumb, and just uses the stuff in the tarball (which could all be different). What you describe is certainly possible with the underlying library.

Perhaps something like --extra_tags=latest,build_by_josh,prod, which could apply to all images in addition to what's specified?

[joshburkart]

Ah, yeah, for a container_bundle, I guess it makes less sense to have the tags be overridable... I'm mostly concerned with container_push -- is there some reason why the semantics of container_push and container_bundle should be linked...?

I guess the behavior I would have expected from container_push would be "push an image, maybe with a default set of tags specified in the build rule, but arbitrarily overridable on the command line", but I'm probably missing some context...

[mattmoor]

Ah, sorry I thought you were talking about the variant of container_push that pushed a bundle (not the bundle rule itself), vs. the singleton container_push. I suppose either could support this.

[joshburkart]

Thanks @mattmoor. Proposal:

1. Near term: deprecate the tag attribute of container_push in favor of a new tags attribute, of type attr.string_list(allow_empty=False).
   • Concrete use case: I'd like the default tags for my pushes to be both latest and built_by_${BUILD_USER}.
2. Longer term/lower priority: Modify the container_push script to accept a list of --extra_tag args as you suggested.
   • Maybe also something like --no_default_tags to optionally ignore the default tags in the build rule... Would have to think this through...

WDYT? Happy to (try to) put together a PR for (1) if you think it's a good path forward...

[mattmoor]

I think my bias would be to stick with 2., since 1. is possible already with container_bundle and it's container_push (should be a simple skylark macro e.g.).

[mattmoor]

Do you want to add something to contrib/?

[joshburkart]

Whoops, sorry, forgot to respond to this! You're right -- container_bundle worked for me for (1). For now I think (2) is lower priority, so I think I'm good to go for now! Thanks for your help @mattmoor.

[kwiesmueller]

Is there an ETA on this? We started trying bazel for our current projects and this is slightly blocking us as we already have to define container_push for every image we want in our cloudbuild.yaml and having to duplicate all this to get latest AND version would be really annoying.

Or what is the best practice for building and pushing multiple images anyways? It does seem odd to me, that it's necessary to use bazel run for every image instead of doing bazel build //... to build and push all of them.

[fejta]

Over at kubernetes we use a macro that allows us to do something like tags(**{'gcr.io/k8s-prow/foo': '//frobber'}) which will build the //frobber image and tag it with gcr.io/k8s-prow/foo:latest, gcr.io/k8s-prow/foo:20180601-deadbeef and gcr.io/k8s-prow/foo:latest-fejta

https://github.com/kubernetes/test-infra/blob/be0a3a082ee35f44b0cc75f45e4f3a068e529664/image.bzl#L24-L39

We then send these values to our container_bundle() target:

https://github.com/kubernetes/test-infra/blob/be0a3a082ee35f44b0cc75f45e4f3a068e529664/prow/BUILD.bazel#L12-L33

So we have a bunch of go_image() targets, which we then put into a single container_bundle() targets that actually builds, tags and pushes all these images.

[mattmoor]

bazel build should be hermetic, so it doesn't talk to anything external (this facilitates distributed build execution in a network jail to enforce hermeticity).

bazel run can talk to whatever it wants.

There is a version of docker_push that takes a docker_bunde and can push as many images as you want in a single execution.

[kwiesmueller]

Thanks @fejta , will check this out. How do you proceed after the bundle step?

I can't find any results generated from for example:

container_bundle(
    name = "bundle",
    images = {
        "$(registry)/$(project)/listener:latest": "//cmd/listener:image",
        "$(registry)/$(project)/listener:$(version)": "//cmd/listener:image",
    },
)

that look like bundle.tar and going furher did not find a way yet to push all those to for example google container registry

[kwiesmueller]

@mattmoor GitHub did not refresh the page to show your comment... I understand the build restriction, that makes sense.

you are talking about docker_push, so far I only saw container_push, am I on a wrong track? And where would this multi push be?

[mattmoor]

See here for an example. docker_push is container_push with the image kind hard-coded to Docker.

[kwiesmueller]

That's awesome thx! At least the cloud builder is now working. That helped a lot. Locally i encounter this python3 vs. python2 error when pushing, but that seems tracked already.

[fejta]

How do you proceed after the bundle step?

Yep at mattmoor mentions we define a docker_push() aka container_push() rule to push the bundle: container_push(name="release-push", bundle="//path/to/my:bundle")

https://github.com/kubernetes/test-infra/blob/be0a3a082ee35f44b0cc75f45e4f3a068e529664/prow/BUILD.bazel#L35-L38

Then I can build, tag and push the images with bazel run //whatever:release-push