I noticed that the remaining test that we have in environment.bats still appears to be affected by the same issue about "properly goes red on CI but stays green when run locally" (the one we solved on the previous PR by moving the "executable bit" test to Ruby / rspec).
In other words, running make buildkite-plugin-test will always succeed locally even if some commands are supposed have which respond in failure, while it will fail as expected on CI.
Failed Attempt at Fixing
At that point that made me wonder if we should keep the plugin-tester thing vs moving everything to Ruby tests. But then when attempting to do that, I noticed that even when running the test in Ruby (expect(system(%Q(/bin/bash -c 'source "$PWD/hooks/environment" >/dev/null; command -v "#{cmd}"'))).to be true) always passes locally, so this time the problem isn't solved by migrating to Ruby.
But after running docker run -it … to test the commands interactively in the container (see spoiler blocks below), I ended up confirming that running which (as well as command -v) in those containers locally (from my Mac) always returns the path to files that can be found in $PATH… even when those are confirmed to not have the executable bit. While those same commands fail on CI for the same case.
🕵️ Apparently `which` (and `command -v`) don't care about the executable bit when run on Docker Mac locally (while they do on CI)
```
$ docker run -it --rm -v "/Users/olivierhalligon/Documents/Dev/tools/a8c-ci-toolkit-buildkite-plugin":/plugin -w /plugin ruby:2.7.4 /bin/bash
root@b25828ad31a5:/plugin# which save_cache
root@b25828ad31a5:/plugin# which cache_repo
root@b25828ad31a5:/plugin# source hooks/environment
~~~ :file_cabinet: Loaded Caching Plugin
root@b25828ad31a5:/plugin# which save_cache
./bin/save_cache
root@b25828ad31a5:/plugin# which cache_repo
./bin/cache_repo
root@b25828ad31a5:/plugin# ls -l $(which cache_repo)
-rw-r--r-- 1 root root 496 Mar 2 11:11 ./bin/cache_repo
```
🕵️ The same is true when I try to run it in the container used to run bats
```
$ docker run -it --rm -v "${PWD}":/plugin buildkite/plugin-tester /bin/bash
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
bash-5.1# which cache_repo
bash-5.1# which save_cache
bash-5.1# source hooks/environment
~~~ :file_cabinet: Loaded Caching Plugin
bash-5.1# which save_cache
./bin/save_cache
bash-5.1# which cache_repo
./bin/cache_repo
bash-5.1# ls -l $(which cache_repo)
-rw-r--r-- 1 root root 496 Mar 2 11:11 ./bin/cache_repo
```
➡️ So it's really the which / command -v implementations that differs when running that docker container from a Mac vs from CI… 😒
What's left in this PR
I ended up bailing on this for now and I guess we'll have to consider living with the fact that those "check that all commands are exposed in $PATH" tests will never to be trusted when running make test locally and will never behave the same on local vs CI. So in the end I didn't change anything in that part…
But, while I was working on this, I made some tiny improvements on the Ruby test about executable bit to make it more Ruby-esque… so I ended up commiting that change anyway, and that's all what that PRs ends up being after all.
Why?
I noticed that the remaining test that we have in
environment.batsstill appears to be affected by the same issue about "properly goes red on CI but stays green when run locally" (the one we solved on the previous PR by moving the "executable bit" test to Ruby / rspec).In other words, running
make buildkite-plugin-testwill always succeed locally even if some commands are supposed havewhichrespond in failure, while it will fail as expected on CI.Failed Attempt at Fixing
At that point that made me wonder if we should keep the
plugin-testerthing vs moving everything to Ruby tests. But then when attempting to do that, I noticed that even when running the test in Ruby (expect(system(%Q(/bin/bash -c 'source "$PWD/hooks/environment" >/dev/null; command -v "#{cmd}"'))).to be true) always passes locally, so this time the problem isn't solved by migrating to Ruby.But after running
docker run -it …to test the commands interactively in the container (see spoiler blocks below), I ended up confirming that runningwhich(as well ascommand -v) in those containers locally (from my Mac) always returns the path to files that can be found in$PATH… even when those are confirmed to not have the executable bit. While those same commands fail on CI for the same case.🕵️ Apparently `which` (and `command -v`) don't care about the executable bit when run on Docker Mac locally (while they do on CI)
``` $ docker run -it --rm -v "/Users/olivierhalligon/Documents/Dev/tools/a8c-ci-toolkit-buildkite-plugin":/plugin -w /plugin ruby:2.7.4 /bin/bash root@b25828ad31a5:/plugin# which save_cache root@b25828ad31a5:/plugin# which cache_repo root@b25828ad31a5:/plugin# source hooks/environment ~~~ :file_cabinet: Loaded Caching Plugin root@b25828ad31a5:/plugin# which save_cache ./bin/save_cache root@b25828ad31a5:/plugin# which cache_repo ./bin/cache_repo root@b25828ad31a5:/plugin# ls -l $(which cache_repo) -rw-r--r-- 1 root root 496 Mar 2 11:11 ./bin/cache_repo ```🕵️ The same is true when I try to run it in the container used to run
``` $ docker run -it --rm -v "${PWD}":/plugin buildkite/plugin-tester /bin/bash WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested bash-5.1# which cache_repo bash-5.1# which save_cache bash-5.1# source hooks/environment ~~~ :file_cabinet: Loaded Caching Plugin bash-5.1# which save_cache ./bin/save_cache bash-5.1# which cache_repo ./bin/cache_repo bash-5.1# ls -l $(which cache_repo) -rw-r--r-- 1 root root 496 Mar 2 11:11 ./bin/cache_repo ```bats➡️ So it's really the
which/command -vimplementations that differs when running that docker container from a Mac vs from CI… 😒What's left in this PR
I ended up bailing on this for now and I guess we'll have to consider living with the fact that those "check that all commands are exposed in $PATH" tests will never to be trusted when running
make testlocally and will never behave the same on local vs CI. So in the end I didn't change anything in that part…But, while I was working on this, I made some tiny improvements on the Ruby test about executable bit to make it more Ruby-esque… so I ended up commiting that change anyway, and that's all what that PRs ends up being after all.
To Test
chmod -x bin/cache_repo, then runmake testlocally and validate it goes redgit restore bin/cache_repoand validate that runningmake testlocally passes and goes back green