search

rocker-org / rocker-versioned

Run current & prior versions of R using docker
https://hub.docker.com/r/rocker/r-ver
GNU General Public License v2.0
297 stars 169 forks source link

Aliases for latest point releases of each version #42

Closed bhaskarvk closed 7 years ago

bhaskarvk commented 7 years ago

It might be helpful to have aliases for latest point releases of each version. So something akin to 3.1.latest, 3.2.latest.... 3.4.latest.

Also stable which points to latest 3.x.latest (3.4.1 currently), and oldrel which points to latest point release of current -1 release.

Also 3.2.5 is missing from the builds.

nuest commented 7 years ago

I don't think the following extension is feasible, but it would surely be fun to use the release nick names, too: https://stackoverflow.com/questions/13478375/is-there-any-authoritative-documentation-on-r-release-nicknames

However, imho if you want to use a specific minor release, then you might just as well even fix the bugfix version.

bhaskarvk commented 7 years ago

That's the thing, I don't want to use a specific minor release. I just want to use the latest minor release of a release. so 3.4.latest points to 3.4.1, 3.3.latest points to 3.3.3 and they get updated as a new minor release is pushed.

nuest commented 7 years ago

We might have a misunderstanding, I was using the <major>.<minor>.<bugfix> pattern for versions. Then you're saying you don't want a specific bugfix release.

In my understanding, that is what 3.1 already does, so 3.1 == 3.1.latest.

bhaskarvk commented 7 years ago

rats, my bad. I should've read more correctly.

cboettig commented 7 years ago

@bhaskarvk @nuest Note that we don't have such tags, e.g. docker pull rocker/r-ver:3.2 will throw an error.

Aliasing Docker tags in automated builds isn't supported by docker hub, https://stackoverflow.com/questions/35792027/dockerhub-set-tag-alias-in-automated-build

This can be done if the docker image is built locally and the same image is pushed to the hub using different tags (e.g. 3.2.3 and 3.2), but any hub setup would result in too separate automated builds. That means these images would have different hashes, even though they were effectively the same, which is obviously not desirable. (confuses users, defeats the layer cache pull system, etc).

Note that while official images do this aliasing (e.g. https://hub.docker.com/r/library/mysql/, see ruby, python etc as well) since they are not built by hub directly, even those images often have only one minor version per minor version anyway. Here we want to stick to automated builds both for the convenience and the transparency/trust element it provides.

nuest commented 7 years ago

I completely support the goal of having automated builds. I've tried a solution using a post_push build hook, see https://docs.docker.com/docker-cloud/builds/advanced/#override-build-test-or-push-commands.

Changes are here: https://github.com/rocker-org/rocker-versioned/compare/master...nuest:master

The tags get created, see https://hub.docker.com/r/nuest/rocker-versioned/tags/

Regarding the image hashes. The tags are all shown for one image, and are the same.

$ docker inspect nuest/rocker-versioned:3
[
    {
        "Id": "sha256:c284ee17661b80d1bc12428714a09111d7b8bb0efa7f9962ed7659f12d021d9a",
        "RepoTags": [
            "nuest/rocker-versioned:3",
            "nuest/rocker-versioned:3.4",
            "nuest/rocker-versioned:3.4.1"
        ],
        "RepoDigests": [
            "nuest/rocker-versioned@sha256:aba2f20d9c91ae2efca1a0567198ddf5badfbc97c842198a4bffd4651e80f749"
        ],

[...]
$ docker images --no-trunc | grep nuest/rocker
nuest/rocker-versioned              3                        sha256:c284ee17661b80d1bc12428714a09111d7b8bb0efa7f9962ed7659f12d021d9a   20 minutes ago      577MB
nuest/rocker-versioned              3.4                      sha256:c284ee17661b80d1bc12428714a09111d7b8bb0efa7f9962ed7659f12d021d9a   20 minutes ago      577MB
nuest/rocker-versioned              3.4.1                    sha256:c284ee17661b80d1bc12428714a09111d7b8bb0efa7f9962ed7659f12d021d9a   20 minutes ago      577MB

In fact, the inspect output is completely the same: diff <(echo $(docker inspect nuest/rocker-versioned:3)) <(echo $(docker inspect nuest/rocker-versioned:3.4.1))

What do you think: Would such a post_push hook be suitable for r-ver? I have no idea how this impacts your current workflow, and if it might increase the work.

cboettig commented 7 years ago

@nuest Very cool.

Looks like this is a works on a push hook though, while for rocker-versioned images build-on-push is actually turned off (would trigger ~ 30 different images being built), and images build when a CRON job posts the appropriate trigger for the appropriate image. (e.g. latest, devel tags are built nightly, other tags are built monthly). Perhaps this just needs to be moved to a build directory and then it would work though.

Another question, do these work on Docker Hub or just Docker Cloud? To be honest I haven't really migrated to Docker Cloud or quite wrapped my head around it. I get that it's supposed to replace Docker Hub, and I presume things will get automatically migrated from the Hub (e.g. all the rocker repos are now showing up on Cloud as well, though afaik the automated builds are occurring only on the Hub. Not sure if/when those will be migrated).

nuest commented 7 years ago

Re. Docker Cloud: I must admit I've not even looked at Docker Cloud yet either, so the approach shown works on Docker Hub even though I sent a link to Docker Cloud documentation. I have no info about migration or discontinuation of Cloud vs. Hub... and didn't find any. I guess it's wait and see.

Re. push hook: For the test above I manually triggered the build in the Docker Hub Build Settings page. I just tested with a build triggered via API (default config, $ curl -H "Content-Type: application/json" --data '{"build": true}' -X POST https://registry.hub.docker.com/u/nuest/rocker-versioned/trigger/e1f...) and it worked, no changes to the build command made in my case.

@cboettig Can you point me to the CRON job definitions? I could test a bit further if specific triggers make a difference.

Of course, once you add a build hook, too, there might be things that could be automated which your are now doing manually (e.g. in the local Makefile), but I have to better understand your workflow to suggest anything there.

cboettig commented 7 years ago

@nuest Nice! Great that this works on Hub, and thanks for figuring this out.

Ah, right, I think I was just being misled by the parallel language between GitHub and DockerHub, this is post_push hook, which I believe refers to docker push, pushing the image the Docker registry, nothing to do with git commit / git push. So yeah, should do exactly what we need. Can you send a PR? Would be good to document what this is doing (e.g. linking to that documentation you provided above) so others can follow. Thanks much!

I've tried to make our build process relatively simple / transparent, but could clearly do better. The Makefiles were basically a crutch for me to sync changes to Dockerfiles in all the versions while there was still quite a bit of active development in exactly what software was in each image. At this point the Dockerfiles for tags <= 3.4.0 are pretty stable and we don't actually want to sync changes across them, so Makefile is just syncing changes to latest and 3.4.1.

Ideally we should be able to get rid of the Makefiles entirely, and really we should be using these post hooks to make 3.4.1 the same as latest. (currently I had to have them as separate Dockerfiles anyway since I didn't have this post hook, which also lead me to triggering them on a separate CRON schedule. Ideally when a new version is released, we should be able to just update the push hook and add the new Dockerfile, and not have to edit the CRON triggers manually).

Re CRON jobs, right, they are just running those curl POST commands the hub shows on the build page. I won't paste the whole crontab here since it includes that access credential, but I'd be happy to add you to the rocker-versioned docker hub repos. Here's the basic template, note the first block is monthly builds using build: true, the second block is the nightly builds for specific tags. Happy for any suggested improvements / streamlining of this. 

## Monthly build all of rocker-versioned
0 0 1 * * curl -H "$Content-Type: application/json" --data '{"build": true}' -X POST https://registry.hub.docker.com/u/rocker/r-ver/trigger/...
1 0 1 * * curl -H "Content-Type: application/json" --data '{"build": true}' -X POST https://registry.hub.docker.com/u/rocker/rstudio-stable/trigger/...

## Nightly build latest, devel tags
0 0 * * * curl -H "Content-Type: application/json" --data '{"docker_tag": "devel"}' -X POST https://registry.hub.docker.com/u/rocker/r-ver/trigger/...
1 0 * * * curl -H "Content-Type: application/json" --data '{"docker_tag": "latest"}' -X POST https://registry.hub.docker.com/u/rocker/...
nuest commented 7 years ago

Re. removing dependencies on Makefiles: How should we detect which version is the "latest"? We could maintain this information explicitly somewhere in the GH repo (is is available during post_push) ... or just assume that the directory with the highest version number is latest.

I tried out the latter approach (see https://github.com/rocker-org/rocker-versioned/compare/master...nuest:master) - just ls "parent dir" | grep "starting with number regex" | sort --version-sort | tail -n 1 to get the last directory), and it worked (see code change: ). Docker Cloud has an adjustable build context but Docker Hub does not, so another approach might be better in the (distinct?) future. Side note: I encountered the really cool --version-sort when testing with directories named 3.10.0 and 3.10.1 - it's a life saver here!

Pulling the latest image and then all the tags will only do one actual download:

$ docker pull nuest/rocker-versioned
Using default tag: latest
latest: Pulling from nuest/rocker-versioned
c75480ad9aaf: Already exists 
078be078f1b1: Pull complete 
Digest: sha256:94f6434a26da4e356c40f72df79b6b40e52b06a7e0d5c404bf4ab795f84dc734
Status: Downloaded newer image for nuest/rocker-versioned:latest

$ docker pull nuest/rocker-versioned:3
3: Pulling from nuest/rocker-versioned
Digest: sha256:94f6434a26da4e356c40f72df79b6b40e52b06a7e0d5c404bf4ab795f84dc734
Status: Downloaded newer image for nuest/rocker-versioned:3

$ docker pull nuest/rocker-versioned:3.4
3.4: Pulling from nuest/rocker-versioned
Digest: sha256:94f6434a26da4e356c40f72df79b6b40e52b06a7e0d5c404bf4ab795f84dc734
Status: Downloaded newer image for nuest/rocker-versioned:3.4

$ docker pull nuest/rocker-versioned:3.4.1
3.4.1: Pulling from nuest/rocker-versioned
Digest: sha256:94f6434a26da4e356c40f72df79b6b40e52b06a7e0d5c404bf4ab795f84dc734
Status: Downloaded newer image for nuest/rocker-versioned:3.4.1

$ docker pull nuest/rocker-versioned:latest
latest: Pulling from nuest/rocker-versioned
Digest: sha256:94f6434a26da4e356c40f72df79b6b40e52b06a7e0d5c404bf4ab795f84dc734
Status: Image is up to date for nuest/rocker-versioned:latest

Re. CRON jobs: no suggestions, seems simple enough!

Will work on the PR, probably early next week. Would you add these tags also to the downstream images (rverse/rstudio)?

cboettig commented 7 years ago

Nice. Assuming the highest version number is also latest seems reasonable to me.

Right, we'd want consistent tag behavior across the entire stack, so it would make sense to just put a copy of these hooks/post_push files next to each Dockerfile, for every file in the stack. (Though I saw your comment about how on Docker Cloud you can specify a different location for the build context vs the Dockerfile location, were you thinking we could then just configure all Dockerfiles to use the same context directory containing a the hooks/post_push file so we don't have to make copies of it?

nuest commented 7 years ago

Yes, basically we would have to put a hooks/post_push next to each Dockerfile.

I do not know if how changing the context affects the hooks directory, and neither did I find any documentation, so I asked in the forum.

Besides the open technical question: Afaik Docker Cloud might switch to a paid model in the future, so should Rocker continue to focus on Docker Hub?

cboettig commented 7 years ago

Sounds good. I gather Docker Cloud is the re-envisioning of how Docker is supposed to make money and that Docker Hub will be completely phased out in place of Docker Cloud. (Development of Docker Hub has been at halted for some time now; issues/bug-reports requesting features on Hub like the ability to migrate a repo to a new host are marked wontfix, wait for Docker Cloud). Docker Hub was obviously trying to be the GitHub hybrid model: public data free, pay for private data, but on Cloud seems to be predicated on making money by selling you actual cloud compute time, e.g. the AWS model. I think they will/do have fee-based containers, but imagine they'll always want to host plenty of free containers like Rocker, but make it easy for users to deploy a Rocker container on their cloud (as opposed to people paying digital ocean or AWS to run the container). Clearly just my speculating here.

nuest commented 7 years ago

I concur with your speculations. If they would make automated build a paid feature, they would shoot themselves in the foot. I have not gotten a reply on the question of hooks vs. context, so we should be fine getting things to work the Docker Hub way for now and read the documentation in a strict sense ("... hooks in your source code repository at the same directory level as your Dockerfile").

Apologies for the delay, it's really just replicating the file, but I want to run some tests with my fork before opening the PR to make sure this works with the different stacks. Here's my checklist, comments welcome:

cboettig commented 7 years ago

sounds like a plan

nuest commented 7 years ago

r-ver images are working now in my fork, for other images see links in my comment above. Had to make some fixes for major and major.minor tags to work properly. All of that took quite long, since I made some stupid mistakes that the build on Hub only told me about after waiting for a few hours (each time).

I've added some logging output, see for example this build for 3.4.1 which receives all possible tags (3, and 3.4 and latest)

...
Removing intermediate container 1dab833a2a7d
Successfully built 7f698b03313c
Pushing index.docker.io/nuest/rocker-versioned:3.4.1...
Done!
Executing post_push hook...
Add version tags for major and major.minor releases.
This version is also the newest major.minor version, add tag '3.4' to '3.4.1'
The push refers to a repository [docker.io/nuest/rocker-versioned] 0b7da705d1dc: Preparing cf4ecb492384: Preparing
cf4ecb492384: Layer already exists
0b7da705d1dc: Layer already exists
3.4: digest: sha256:9612c3c66c407c5c747bddafc86a32cb1a386ef4d7337b529aab83529d13a604 size: 742
This version is also the newest major version, add tag '3' to '3.4.1'.
The push refers to a repository [docker.io/nuest/rocker-versioned]
0b7da705d1dc: Preparing
cf4ecb492384: Preparing
cf4ecb492384: Layer already exists
0b7da705d1dc: Layer already exists
3: digest: sha256:9612c3c66c407c5c747bddafc86a32cb1a386ef4d7337b529aab83529d13a604 size: 742
This version is also newest version overall, add 'latest' tag to 3.4.1
The push refers to a repository [docker.io/nuest/rocker-versioned]
0b7da705d1dc: Preparing
cf4ecb492384: Preparing
cf4ecb492384: Layer already exists
0b7da705d1dc: Layer already exists
latest: digest: sha256:9612c3c66c407c5c747bddafc86a32cb1a386ef4d7337b529aab83529d13a604 size: 742
Version tagging completed.
Build finished

@cboettig two questions:

PR #43 is ready for review.

nuest commented 7 years ago

@bhaskarvk My suggestion does not use the oldrel and stable tags you originally suggested. Are your requirements served with tags like 3 and 3.3 (implying the "latest" of missing numbers, i.e. 3 = 3.latest.latest) ?

cboettig commented 7 years ago

binary is basically an earlier attempt at versioned images that could potentially go farther back by using substantially older Debian releases. This basically works (though some in-between versions would still need to fiddling to install), but opens up other issues; in general really old versions of R is a somewhat niche exercise that doesn't fit into the rest of the stack here, so just ignore it. Currently docker hub isn't building any of those Dockerfiles.

Will be away from email for some conference travel etc but I'll take a look at the PR on the other side in a few weeks!

nuest commented 7 years ago

@cboettig From https://github.com/rocker-org/rocker-versioned/commit/e663d90ce12316e26bc56e6410cad24918bef104 I understand that things are not working as wished or expected. I'd like to fix that! I probably do not fully grasp the interactions between the hooks and cron jobs. Can you briefly describe the issue you encountered?

bhaskarvk commented 7 years ago

@nuest I think I am good.

cboettig commented 7 years ago

@nuest, I think everything is working now as expected. The cron issue was entirely my mistake, (I had missing quotes in my json data so it wasn't matching the latest tag, but just building everything).

You should be able to pull tags at any level of precision, e.g. rocker/r-ver:3, rocker/r-ver:3.4, rocker/r-ver:3.4.1, as expected and get the right version now. Should be propagated up through verse at least, guess we can also add the hooks to geospatial.

bhaskarvk commented 7 years ago

Hooks to geospatial will be much appreciated.

nuest commented 4 years ago

The documentation about the major/minor version tags was removed in https://github.com/rocker-org/rocker-versioned/commit/66cf4e12b38ec99b7118b4df10d927eba9ecd172#diff-04c6e90faac2675aa89e2176d2eec7d8 and with https://github.com/rocker-org/rocker-versioned/commit/e303edf1a87f771e197f49acd6eb8151c63f360a the docs were commented out in VERSIONS.md because "(tags may not always be available)".

Looking at the tags on Docker Hub, it looks like (at the point in time of the screenshot below) 3 and 3.6 are equal to 3.6.0, which is fine because 3.6.1 is currently the latest and is not yet using a time-stamped MRAN, which is what you would expect with version tagged releases.

image

@cboettig maybe you can elaborate futher on the issue?

I'm happy to revisit this feature to make it work!