env dependency on image triggers constant rebuild

[Enteee]

I am using an unrelease dobi version built from master containing the fix for #141: dobi version 0.11.1 (build: 15cb0b6, date: Mon Jul 23 13:13:27 UTC 2018

When an image resource depends on an env resource, the image resource is constantly rebuilt. After fixing #141 this has a big impact on the rest of the build chain since image rebuilds now correctly trigger rebuilds of all the other resources.

tree

.
├── A
│   ├── a
│   └── Dockerfile
└── dobi.yaml

cat dobi.yaml

env=env/base:
  variables: [
    "A=A"
  ]

image=image/A:
  depends: [
    "env/base"
  ]
  image: "a"
  context: "A"
  tags: [
    "latest",
  ]

A/Dockerfile:

FROM alpine

The first time i run dobi image/A builds the image a

$ dobi image/A
[WARN] meta.project is not set. Using default "dobi-bug".
[env:set env/base] Done
Step 1/1 : FROM alpine
 ---> 3fd9065eaf02
Successfully built 3fd9065eaf02
Successfully tagged a:latest
[image:build image/A] a Created

every subsequent call do dobi image/A triggers a rebuild of a

$ dobi image/A
[WARN] meta.project is not set. Using default "dobi-bug".
[env:set env/base] Done
Step 1/1 : FROM alpine
 ---> 3fd9065eaf02
Successfully built 3fd9065eaf02
Successfully tagged a:latest
[image:build image/A] a Created

This is reasonably fast, since docker caches build steps. But it triggers job resource with dependency on this image to run again.

[Enteee]

And an example with an actual job attached: tree:

.
├── a
├── A
│   └── Dockerfile
└── dobi.yaml

dobi.yaml:

env=env/base:
  variables: [
    "A=A"
  ]

mount=mount/.:
  bind: "."
  path: "/out"

image=image/A:
  depends: [
    "env/base"
  ]
  image: "a"
  context: "A"
  tags: [
    "latest",
  ]

job=A:
  use: "image/A"
  command: "touch /out/a"
  artifact: [
    "a"
  ]
  mounts: [
    "mount/."
  ]

A/Dockerfile:

FROM alpine

every time you run dobi A the job A is run:

$ dobi  A
[WARN] meta.project is not set. Using default "dobi-bug".
[env:set env/base] Done
Step 1/1 : FROM alpine
 ---> 3fd9065eaf02
Successfully built 3fd9065eaf02
Successfully tagged a:latest
[image:build image/A] a Created
[job: A] touch /out/a -> a Start
[job: A] touch /out/a -> a Done

[dnephin]

Ah, this is actually the expected behaviour, but I can see how it's not ideal.

Right now the "stale vs fresh" logic for an environment variable is "has the environment changed". So setting A=A is changing the environment and everything following it will be considered stale because the env task marked itself as modified. This is similar to how make will rebuild everything if a dependency changes.

It's possible that the env resource could always return "not modified", but I don't think that would be correct for a job task. To handle this properly we would need to record the full environment for every task and use that to determine if there was a change.

As I mentioned in https://github.com/dnephin/dobi/issues/141#issuecomment-407275112 to handle a lot of these edge cases properly it might be necessary to rebuild some parts of dobi using buildkit.