Custom builder should provide input for hash calculation for caching

Workaround: pass dummy args to the build script that serve to differntiate cache calculation inputs on an otherwise equal custom build config (gotcha: the image name is not taken into account). Or — better — split build instructions into separate files and watch them separatly (with the added benefit that only one image cache is invalidated -as oposed to all images of this file)

I delete all images from local docker store
I invalidate the skaffold cache rm ~/.skaffold/cache

Then I build with skaffold dev, I get this result in docker store:

ae-dir/web2ldap                                            19553c75108c03ec178835cf4edf464193af58453fe01435ecbd48cbcc6ba285   19553c75108c        About a minute ago   182MB
ae-dir/web2ldap                                            65ab89b                                                            19553c75108c        About a minute ago   182MB
ae-dir/web-server                                          c4bbb592d9ec914977a6e85995fc83fcad8b0123decf0affe7b409730a5f4d5c   c4bbb592d9ec        About a minute ago   77.9MB
ae-dir/web-server                                          65ab89b                                                            c4bbb592d9ec        About a minute ago   77.9MB

Tags used in deployment:
- ae-dir/config-initializer -> ae-dir/config-initializer:b3b2a891f7bab54c5c41dd8dd6b1f6fdeff55af67081bc342c0b7acba12a48be
- ae-dir/openldap-ms -> ae-dir/openldap-ms:20d02c5b6af09e79f92857c49aa2cef8b7015c78a6520f1bb9da1bd74bf3a1f4
- ae-dir/web-server -> ae-dir/web-server:c4bbb592d9ec914977a6e85995fc83fcad8b0123decf0affe7b409730a5f4d5c
- ae-dir/web2ldap -> ae-dir/web2ldap:19553c75108c03ec178835cf4edf464193af58453fe01435ecbd48cbcc6ba285
Loading images into k3d cluster nodes...
- ae-dir/config-initializer:b3b2a891f7bab54c5c41dd8dd6b1f6fdeff55af67081bc342c0b7acba12a48be -> Loaded
- ae-dir/openldap-ms:20d02c5b6af09e79f92857c49aa2cef8b7015c78a6520f1bb9da1bd74bf3a1f4 -> Loaded
- ae-dir/web-server:c4bbb592d9ec914977a6e85995fc83fcad8b0123decf0affe7b409730a5f4d5c -> Loaded
- ae-dir/web2ldap:19553c75108c03ec178835cf4edf464193af58453fe01435ecbd48cbcc6ba285 -> Loaded
Images loaded in 57.452258464s

Then I stop skaffold and a run skaffold dev again, I get this result in docker store:

ae-dir/web-server                                          19553c75108c03ec178835cf4edf464193af58453fe01435ecbd48cbcc6ba285   19553c75108c        17 minutes ago      182MB
ae-dir/web-server                                          65ab89b                                                            19553c75108c        17 minutes ago      182MB
ae-dir/web2ldap                                            19553c75108c03ec178835cf4edf464193af58453fe01435ecbd48cbcc6ba285   19553c75108c        17 minutes ago      182MB
ae-dir/web2ldap                                            65ab89b                                                            19553c75108c        17 minutes ago      182MB

Checking cache...
- ae-dir/config-initializer: Found Locally
- ae-dir/openldap-ms: Found Locally
- ae-dir/web-server: Found. Tagging
- ae-dir/web2ldap: Found Locally
Tags used in deployment:
- ae-dir/config-initializer -> ae-dir/config-initializer:b3b2a891f7bab54c5c41dd8dd6b1f6fdeff55af67081bc342c0b7acba12a48be
- ae-dir/openldap-ms -> ae-dir/openldap-ms:20d02c5b6af09e79f92857c49aa2cef8b7015c78a6520f1bb9da1bd74bf3a1f4
- ae-dir/web-server -> ae-dir/web-server:19553c75108c03ec178835cf4edf464193af58453fe01435ecbd48cbcc6ba285
- ae-dir/web2ldap -> ae-dir/web2ldap:19553c75108c03ec178835cf4edf464193af58453fe01435ecbd48cbcc6ba285
Loading images into k3d cluster nodes...
- ae-dir/config-initializer:b3b2a891f7bab54c5c41dd8dd6b1f6fdeff55af67081bc342c0b7acba12a48be -> Loaded
- ae-dir/openldap-ms:20d02c5b6af09e79f92857c49aa2cef8b7015c78a6520f1bb9da1bd74bf3a1f4 -> Loaded
- ae-dir/web-server:19553c75108c03ec178835cf4edf464193af58453fe01435ecbd48cbcc6ba285 -> Loaded
- ae-dir/web2ldap:19553c75108c03ec178835cf4edf464193af58453fe01435ecbd48cbcc6ba285 -> Loaded

That means skaffold retags an image to the other. Consequently the other images gets pulled into the deployment and boom.

I do a custom build:

set -e

repo=$(echo "$IMAGE" | cut -d':' -f 1)
tag=$(echo "$IMAGE" | cut -d':' -f 2)
out="$(mktemp -u)"
nix-build "$BUILD_CONTEXT/Nixfile.nix" -A "images.$repo" -o "$out" --arg image-tag "\"$tag\""
${docker}/bin/docker load -i $out

if $PUSH_IMAGE; then
  ${docker}/bin/docker push $IMAGE
fi

Clearly the part that's wrong here is:

Checking cache...
...
 - ae-dir/web-server: Found. Tagging
...

After the first build only three cache entries are construed for four images:

       │ File: /home/blaggacao/.skaffold/cache
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ 5af0136c849475289159f6f370b20ed5a51ee65a31ee8406d2229a12810bc1a7:
   2   │   id: sha256:20d02c5b6af09e79f92857c49aa2cef8b7015c78a6520f1bb9da1bd74bf3a1f4
   3   │ 60a43dba56e2ed8b8d3294fbb23601ba50eb070364324f36859983a21da50198:
   4   │   id: sha256:19553c75108c03ec178835cf4edf464193af58453fe01435ecbd48cbcc6ba285
   5   │ 85d39cdf191fd3e9eebb616b0cc485a3e3aff2920fb637e16be165c0043805c5:
   6   │   id: sha256:b3b2a891f7bab54c5c41dd8dd6b1f6fdeff55af67081bc342c0b7acba12a48be

Hence, the problem seems building the cache references, probably calculating the hashes.

Looks like the artifact config is relevant

    - image: ae-dir/web-server
      context: images/aedir-ui
      custom:
        buildCommand: skaffold-nix-builder
        dependencies:
          paths:
            - ./*.nix
    - image: ae-dir/web2ldap
      context: images/aedir-ui
      custom:
        buildCommand: skaffold-nix-builder
        dependencies:
          paths:
            - ./*.nix

So it looks like the input missed out by the hashing function is -A "images.$repo" - which is a nix equivalent for buildkit's --target. Somewhere around here: https://github.com/GoogleContainerTools/skaffold/blob/46529503e1e8e754644895ccd06bd6aea8cbc128/pkg/skaffold/build/cache/hash.go#L115-L117 and here https://github.com/GoogleContainerTools/skaffold/blob/dd9fbdbdb559db67c7dc66bf33b781d22b5bae46/pkg/skaffold/schema/v2alpha2/config.go#L711-L716

Workarorund: pass a dummy argument to the build command

      context: images/aedir-ui
      custom:
        buildCommand: skaffold-nix-builder web-server
        dependencies:
          paths:
            - ./*.nix
    - image: ae-dir/web2ldap
      context: images/aedir-ui
      custom:
        buildCommand: skaffold-nix-builder web2ldap
        dependencies:
          paths:
            - ./*.nix

Then we get four hashes:

       │ File: /home/blaggacao/.skaffold/cache
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ 5af0136c849475289159f6f370b20ed5a51ee65a31ee8406d2229a12810bc1a7:
   2   │   id: sha256:20d02c5b6af09e79f92857c49aa2cef8b7015c78a6520f1bb9da1bd74bf3a1f4
   3   │ 79ec48e778c1d97942f86a5cbfb649991dacb716c411749e8d686879cc045b8f:
   4   │   id: sha256:19553c75108c03ec178835cf4edf464193af58453fe01435ecbd48cbcc6ba285
   5   │ 85d39cdf191fd3e9eebb616b0cc485a3e3aff2920fb637e16be165c0043805c5:
   6   │   id: sha256:b3b2a891f7bab54c5c41dd8dd6b1f6fdeff55af67081bc342c0b7acba12a48be
   7   │ 935b594a23db76ba2a93a2e1dcd646b708667452aafce4d98b2fe08b280411a8:
   8   │   id: sha256:c4bbb592d9ec914977a6e85995fc83fcad8b0123decf0affe7b409730a5f4d5c

QED

GoogleContainerTools / skaffold

Custom builder should provide input for hash calculation for caching #4737