ccare
commented
4 years ago Interesting. I'd not appreciated the 6 week cycle until now. Although using latest probably won't achieve what you're hoping for.
I think there are two main problems here:
1) What is nightly depends on the invocation time (and to some extent, the state of the caches) on the build environment at the time of build. Given we currently label an image with the git-sha, builds won't be repeatable and this will make it hard to replicate/test issues in different environments.
2) We want to keep the runtime environment up-to-date. Hence the motivation for using nightly.
However, all of this only applies to building of an image. Not to running it. So unless we manually reimage the container (which isn't idempotent) we won't see any change to the runtime environment (even though the container is based on nightly). So whatever we do, we're going to have to do periodic updates of some frequency, the nightly flag won't help with that in that regard (although it would avoid a code change on re-release).
Our other issue is that images are currently labelled off their git sha, so ideally we would have a 'bump' commit to indicate a new version needed building. At the moment a given image version is 1..1 with a git commit (and optionally a git tag) and maintaining that invariant would be helpful for me (it would allow us to rollback etc).
Just to be clear, the current build process is something like this
- take a git commit
- run the equivalent of
docker buildanddocker pushto make an image - release that image as a test runner (worker nodes will do the equivalent of
docker pulland won't re-pull container versions they've already downloaded) - promote that runner to live (we've been adding UI tooling to the website to help manage this)
We could, I guess, have a script that releases a new version of the container every N weeks. We could introduce automatic fixtures into the process and aim for something between option (1) and option (2). I'm thinking automatic testing, but with a manual smoke test.
The minimal process would, I think, look something like this:
- you (or another maintainer, of course) updates the target version in the Dockerfile
- [ideally we add some kind of CI to the repo, and that goes green]
- We trigger a build. Currently this is done by adding the git-sha of the HEAD commit is added to the exercism.io website, it then builds (including running automatic fixtures if we have them)
- I (but probably in the future, the maintainer) press the 'deploy' button to make the image available on our cluster
- We do a quick smoke test on the new version (run some code on the online website)
- We press the 'promote' to live button
We can do as much as we can to make that loop fast. But I think the fundamental question is how often we should (and practically how much time you have to) go around that loop.
@iHiD - you may have an opinion on this too.
iHiD
coriolinus
Hi @coriolinus,
Please feel free to think about a different way of achieving this... but I can't think of one right now.
To ensure that all builds are repeatable, I think we need to be explicit about versions of base images. To this end, I've tried to replace the docker tags 'nightly' and 'latest' with more stable tags.
The first change I've made is, I think, reasonable. We'll want a specific git-commit to relate to a specific rust language.
It's the versioning of the second base image that frustrates me. The nightly builds have no other versioned tagging, so the only way of achieving stability is to be precise with the SHA digest (which has poor readability). I wondered whether there's another docker image we could use here, but I don't know enough about the Rust ecosystem to understand.
Appreciate your insights.