Force rebuild of docker image when local Dockerfile changes

davidferlay commented 9 months ago

Context

As defined in https://github.com/launchrctl/launchr/blob/main/docs/config.md
When executing an action which uses a Dockerfile localy defined, a Docker image is built (before being ran) following the tag section from action.yaml :
```
tags:
  - my/image:version2
  - my/image:version3
```
which with a real example, would often turns to
```
tags:
  - platform-actions-compare-artifact:latest
  - platform-actions-compare-artifact:1.0.0
```

Observed

Let's consider these steps:

User executes action for the 1st time
Docker image gets built according to Dockerfile
Action command is ran in the freshly built image
User updates Dockerfile
User executes action again
Docker image does not get rebuilt according to updated Dockerfile
Action command in ran in outdated image

Note that if user had consciously updated the tag section of action.yaml after updating Dockerfile, the issue may not have occur, but that requires an additional, cumbersome-ish, manual action from user

Proposition

The tag section from action.yaml seems superfluous:

It may be expected and preferable for user to always use an up to date Docker image when executing an action
launchrctl could generate a hash from the Dockerfile (kind of checksum) and use that hash as image tag
the tag section of action.yaml could disapear entirely and it's mechanism abstracted away from user
If the image tag is not present, it will be built and run to execute the action
That way, user benefits from an always up to date action image (and container environment)
That behavior also favors reproducibility

davidferlay commented 8 months ago

Does not seem to be ok

Can be reproduced with action : test.tar.gz

➜  ski-platform git:(711-plasma-core-contrib-custom) ✗ grep "USER " platform/actions/test/Dockerfile
USER plasma

➜  ski-platform git:(711-plasma-core-contrib-custom) ✗ ./launchr_Linux_x86_64 platform:test -vvv
DEBUG: Starting execution of the action "platform:test" in "docker" environment, command [sh /action/main.sh]
Image "test:latest" doesn't exist locally, building...
INFO: Image "test:latest" doesn't exist locally, building...
Step 1/9 : FROM alpine:3.18
 ---> b541f2080109
Step 2/9 : RUN apk update --no-cache --quiet &&     apk upgrade --no-cache -a
 ---> Using cache
 ---> 99a1764c2f7e
Step 3/9 : RUN apk add --no-cache --quiet git curl diffutils
 ---> Using cache
 ---> 58090ed0f82a
Step 4/9 : ARG USER_ID
 ---> Using cache
 ---> 32378c4337e7
Step 5/9 : ARG GROUP_ID
 ---> Using cache
 ---> a9d4e745289f
Step 6/9 : RUN echo ${USER_ID}
 ---> Using cache
 ---> ec27ced977b0
Step 7/9 : RUN echo ${GROUP_ID}
 ---> Using cache
 ---> 76032574ec1b
Step 8/9 : RUN adduser -D -u $USER_ID -g $GROUP_ID plasma || true
 ---> Using cache
 ---> 328f01dd60b3
Step 9/9 : USER plasma
 ---> Running in 608fdefc84b5
 ---> f97635de8fd4
Successfully built f97635de8fd4
Successfully tagged test:latest
+ whoami
plasma
DEBUG: [hijack] End of stdout
DEBUG: Error resize: Error response from daemon: container 00901b9a1765b469dceacf795a3f6193345f33a7f7899892dbbde48d168df1a4 is not running
INFO: action "platform:test" finished with the exit code 0

➜  ski-platform git:(711-plasma-core-contrib-custom) ✗ vi platform/actions/test/Dockerfile 
➜  ski-platform git:(711-plasma-core-contrib-custom) ✗ grep "USER " platform/actions/test/Dockerfile
USER someotheruser

➜  ski-platform git:(711-plasma-core-contrib-custom) ✗ ./launchr_Linux_x86_64 platform:test -vvv
DEBUG: Starting execution of the action "platform:test" in "docker" environment, command [sh /action/main.sh]
Image "test:latest" doesn't exist locally, building...
INFO: Image "test:latest" doesn't exist locally, building...
Step 1/9 : FROM alpine:3.18
 ---> b541f2080109
Step 2/9 : RUN apk update --no-cache --quiet &&     apk upgrade --no-cache -a
 ---> Using cache
 ---> 99a1764c2f7e
Step 3/9 : RUN apk add --no-cache --quiet git curl diffutils
 ---> Using cache
 ---> 58090ed0f82a
Step 4/9 : ARG USER_ID
 ---> Using cache
 ---> 32378c4337e7
Step 5/9 : ARG GROUP_ID
 ---> Using cache
 ---> a9d4e745289f
Step 6/9 : RUN echo ${USER_ID}
 ---> Using cache
 ---> ec27ced977b0
Step 7/9 : RUN echo ${GROUP_ID}
 ---> Using cache
 ---> 76032574ec1b
Step 8/9 : RUN adduser -D -u $USER_ID -g $GROUP_ID plasma || true
 ---> Using cache
 ---> 328f01dd60b3
Step 9/9 : USER someotheruser
 ---> Running in d85a47653516
 ---> 830ec8a7c6a8
Successfully built 830ec8a7c6a8
Successfully tagged test:latest
+ whoami
plasma
DEBUG: [hijack] End of stdout
DEBUG: Error resize: Error response from daemon: container e3cddeecfc293c3d928897e5ba49311257f783f29b227aea8999c54fdc94d2ca is not running
INFO: action "platform:test" finished with the exit code 0

.plasmactl/actions.sum is generated though : platform:test h1:Xt4RVE6iStoWcpv2goLOz8iwscnRg4vor7gNoHXAEoU=

It looks like it does rebuild image as expected, but does not run action in the correct image

That would explain message DEBUG: Error resize: Error response from daemon: container e3cddeecfc293c3d928897e5ba49311257f783f29b227aea8999c54fdc94d2ca is not running

davidferlay commented 8 months ago

Additional tests show it's working as expected :ok_hand:

updating Dockerfile > .plasmactl/actions.sum is updated > image is rebuilt > a new image is used when action is executed
updating script.sh > .plasmactl/actions.sum is updated > image is rebuilt but end up being same image used (which is fine, since script is mounted)
new option --no-cache works as expected
tag section not needed anymore in action.yalm

Let's also update documentation to remove tag part

Weirdly, executing another action does not get listed in .plasmactl/actions.sum though

davidferlay commented 6 months ago

Working as expected :ok_hand:

# Building new binary from PR branch

➜ plasmactl build -p github.com/launchrctl/launchr@8-force_rebuild_on_change,github.com/launchrctl/compose@latest,github.com/skilld-labs/components-bump@latest -n plasmactl -o plasmactl
Starting building plasmactl
Creating project files and fetching dependencies
(...)
Building plasmactl
Build complete: plasmactl

# Getting test action dirs

➜ mkdir -p .compose/build/platform/actions/
➜ cp -r ~/Sources/launchr/example/envvars/ .compose/build/platform/actions/    

➜ ./plasmactl
Usage:
  plasmactl [flags]
  plasmactl [command]

Actions:
  platform:envvars              envvars: Test passing static or dynamic environment variables to container

Additional Commands:
  build                         Rebuilds application with specified configuration
  completion                    Generate the autocompletion script for the specified shell
  components-bump               Bump versions of updated components
  compose                       Composes filesystem (files & dirs)
  discover                      Discovers available actions in filesystem
  help                          Help about any command
  login                         Logs in to services like git, docker, etc.
  logout                        Logs out from a service

Flags:
  -h, --help                        help for plasmactl
      --keyring-passphrase string   Passphrase for keyring encryption/decryption
  -q, --quiet                       disable stdOut
  -v, --verbose count               log verbosity level, use -vvv DEBUG, -vv WARN, -v INFO
      --version                     version for plasmactl

Use "plasmactl [command] --help" for more information about a command.

# Executing action 1st time -> actions.sum is populated with hash

➜ ./plasmactl platform:envvars                          
Image "envvars:latest" doesn't exist locally, building...
Step 1/2 : FROM alpine:3.18
 ---> b541f2080109
Step 2/2 : RUN echo "ACTION_ENV=${ACTION_ENV}"
 ---> Using cache
 ---> b87ebeaa7c5b
Successfully built b87ebeaa7c5b
Successfully tagged envvars:latest
+ echo 'ACTION_ENV=some_value'
ACTION_ENV=some_value

➜ ll .plasmactl/actions.sum   
-rw------- 1 dferlay dferlay 63 Feb 19 12:12 .plasmactl/actions.sum

➜ cat .plasmactl/actions.sum  
envvars:latest h1:yXi5uB7pWoTolmEhj/7l91oAG5j0EpsE2qpQaHc5IxY=

# Executing action more times without Dockerfile changes -> does not rebuild image

➜ ./plasmactl platform:envvars
Image "envvars:latest" exists locally
+ echo 'ACTION_ENV=some_value'
ACTION_ENV=some_value

➜ ./plasmactl platform:envvars
Image "envvars:latest" exists locally
+ echo 'ACTION_ENV=some_value'
ACTION_ENV=some_value

➜ ./plasmactl platform:envvars
Image "envvars:latest" exists locally
+ echo 'ACTION_ENV=some_value'
ACTION_ENV=some_value

➜ ./plasmactl platform:envvars
Image "envvars:latest" exists locally
+ echo 'ACTION_ENV=some_value'
ACTION_ENV=some_value

# Executing action after changing Dockerfile actions.sum is updated with new hash

➜ vi .compose/build/platform/actions/envvars/Dockerfile 

➜ ./plasmactl platform:envvars                         
Image "envvars:latest" doesn't exist locally, building...
Step 1/3 : FROM alpine:3.18
 ---> b541f2080109
Step 2/3 : RUN echo "ACTION_ENV=${ACTION_ENV}"
 ---> Using cache
 ---> b87ebeaa7c5b
Step 3/3 : RUN echo NEW
 ---> Running in 1c5b97409a89
NEW
 ---> 92c22ed112a9
Successfully built 92c22ed112a9
Successfully tagged envvars:latest
+ echo 'ACTION_ENV=some_value'
ACTION_ENV=some_value

➜ ./plasmactl platform:envvars
Image "envvars:latest" exists locally
+ echo 'ACTION_ENV=some_value'
ACTION_ENV=some_value

➜ ll .plasmactl/actions.sum                            
-rw------- 1 dferlay dferlay 63 Feb 19 12:13 .plasmactl/actions.sum
➜ cat .plasmactl/actions.sum                           
envvars:latest h1:stoVGv93HNncGdYq8S7c3i7pa8rFaZ56Ja+5u1X/9/4=

# Executing action more times without Dockerfile changes -> does not rebuild image

➜ ./plasmactl platform:envvars
Image "envvars:latest" exists locally
+ echo 'ACTION_ENV=some_value'
ACTION_ENV=some_value

davidferlay commented 1 month ago

https://github.com/launchrctl/launchr/pull/20

launchrctl / launchr