search

scheme-containers / monorepo

Generators, website, issues
MIT License
6 stars 1 forks source link

Branch/repository organisation #2

Closed weinholt closed 4 years ago

weinholt commented 4 years ago

The weinholt/scheme-docker project is based on having a single project with one branch per Scheme implementation. The reasons for this were:

For this new project, scheme-containers, maybe it makes more sense to have one project per Scheme implementation. There is still some overhead associated with this, but I think it makes more sense. It would also allow us to have issues and permissions that apply only to a single Scheme implementation.

WDYT, @lassik?

weinholt commented 4 years ago

Now that I tried it, it seems a lot easier with one repo per Scheme. See #1.

lassik commented 4 years ago

Looks much nicer with Loko having its own repo than the old branch mess we had :) The separate branches were difficult to browse on GitHub as well as in local Git checkouts, although they were the right call under the circumstances.

How does Docker Hub require us to organize things on the GitHub side?

Why do some of these names show the schemers/loko prefix while others don't, and how does it decide which image is the default one you get with docker run -it schemers/loko? What is the difference between base and latest?

lassik commented 4 years ago

Also one more important question: Is there any implementation for which people would like to track different minor versions? It would be easy to remember and use names like:

Do some people need to track e.g. Guile 2.1 and Guile 2.2 separately, or is it enough to have a guile:2 that always points at the latest Guile 2.x release?

lassik commented 4 years ago

We should probably establish the separate major version branches for every Scheme implementation as a standard procedure. IMHO it makes the organization really clear, and major versions are usually a natural inflection point for any breaking changes.

For new Scheme implementations, we can start by only adding the latest major version. If some people care about earlier major versions, they can always add them later (as we retroactively added Chicken 4 a while back after we already had Chicken 5).

lassik commented 4 years ago

We're just working on adding more Scheme metadata for Schemedoc and SRFI, so I'm motivated to clean up the files in schemedoc/implementation-metadata and establish a formal schema that is followed by each implementation's metadata file.

weinholt commented 4 years ago

I think that Guile 2.0 (if we add it), 2.2 and 3.0 should be separate. They are quite different in practice. Seems reasonable to start with the major versions and add a minor one if anyone requests it.

Why do some of these names show the schemers/loko prefix while others don't, and how does it decide which image is the default one you get with docker run -it schemers/loko? What is the difference between base and latest?

It guess they were just rebuilt due to new base images and the different entries for the tags is just a discrepancy in whatever script triggered the build. (I'm not sure, but I didn't trigger anything myself.)

scheme-containers/loko currently has all its flies in a loko directory. Does Docker Hub require this layout, or is that just the layout resulting from a git filter-branch on the aggregate scheme-docker repo? If Docker Hub doesn't require a subdirectory, IMHO it would be simpler to have the container variants (in this case, base and debian) directly under the root directory of the repo.

It was just that I pushed the same branch as we already had to the new master branch. It would be easier to let the files be in the root.

It would probably make sense to organize each Scheme implementation into "flavors" from the start. Loko requires it anyway since the base and Debian builds are different, but even for implementations where we currently only have one flavor we may want to add another flavor later (e.g. another major version or different build options). In that case, it'll be easier to adjust and to switch between different implementations' repos if even the default Dockerfile is always in a subdirectory.

I guess Loko is going to be the only one that works as a base image, but the other ones can reasonably be had in Alpine and Debian base images. I'm not sure what naming convention to use, but I guess something that combines the major version and the type of the base image.

Btw, if you want another conundrum, how do we prepare for (or implement) builds for non-amd64 architectures? Docker works on armhf and aarch64 as well. For that I was hoping that Docker Hub did something to help, but I think everyone has to invent their own solution.

lassik commented 4 years ago

I think that Guile 2.0 (if we add it), 2.2 and 3.0 should be separate. They are quite different in practice. Seems reasonable to start with the major versions and add a minor one if anyone requests it.

OK, let's do that. If you already know that we'll need to differentiate minor versions for Guile, we should probably do it from the start.

It was just that I pushed the same branch as we already had to the new master branch. It would be easier to let the files be in the root.

OK, let's have an organization so that each directory directly under the repo root corresponds to a branch/flavor in Docker Hub (I'm probably missing the right technical term here :)

I guess Loko is going to be the only one that works as a base image, but the other ones can reasonably be had in Alpine and Debian base images.

Ah, that's right. Is "base" the usual term "FROM scratch" images? Debian and Alpine are have worked well so far for Schemes that need a distro underneath.

I'm not sure what naming convention to use, but I guess something that combines the major version and the type of the base image.

Btw, if you want another conundrum, how do we prepare for (or implement) builds for non-amd64 architectures? Docker works on armhf and aarch64 as well.

Good to hear that they ARM is supported! I didn't know that.

Let's look at some popular projects and find out what naming conventions they use for their images.

Here's Posgres as one example: https://hub.docker.com/_/postgres?tab=tags They have ARM images, but those don't seem to have a different tag (if that's the right term) from the amd64 images.

The versioned images are named (or "tagged"?) like 9.6.17-alpine. If we prepend the Scheme implementation name to that, we'd get names like:

(Here are Alpine's own Guile packages btw: https://pkgs.alpinelinux.org/packages?name=guile&branch=edge 2.0 only, but ARM is supported.)

Perhaps the Debian images should be the default ones:

We should probably also differentiate between:

Those can be aliases momentarily, but then diverge again as new releases are made. Does Docker Hub provide standard tools of some kind for dealing with container aliases?

lassik commented 4 years ago

I never figured out how the colon works in the image names :) So all our names must be like docker run -it schemers/... because that's the Docker Hub organization. But after the slash we can append:

I guess the part after the colon is used for non-default "flavors" of the image, and the default flavor is chosen by some means.

If we assume that the latest major and minor version, based on Debian, is the most often needed choice, we should probably make that the non-colon version.

So docker run -it schemers/guile would run Guile 3.0.0 on Debian right now.

docker run -it schemers/guile:alpine would run Guile 3.0.0 on Alpine (if that can be done).

docker run -it schemers/guile:2.2 would run the latest Guile 2.2.x on Debian.

docker run -it schemers/guile:2.0-alpine would run the latest Guile 2.0.x on Alpine.

Does this kind of naming convention sound reasonable to you?

For Loko, we could also have Debian as the default for consistency with other implementations. So it would be:

schemers/loko - Loko 0.4.3 on Debian

schemers/loko:base - Loko 0.4.3 with no distro underneath

I guess we could make schemers/loko:0.3 and schemers/loko:0.3-base as well. Is it common for people to compare different 0.x minor versions? It most probably depends on the implementation, since some authors are really modest and keep the version number at 0.x for years and years even though the implementation is very stable. Gauche isn't even at 1.0 yet.

lassik commented 4 years ago

Is the part between the slash and the colon always the image name? In that case, we'd best have a GitHub repo with that same name for each implementation. So far things seem simpler than I feared. That means there are probably some surprises waiting :)

lassik commented 4 years ago

I'll add a Chicken repo under this organization to experiment with the above layouts and whether they work as easily as thought. We can replace the repo contents with things generated from the S-expression files once we figure out how to automate the generation as well as making the right commits on GitHub.

lassik commented 4 years ago

If there's something that won't work with the above or if you have better ideas, please tell. I have almost no idea what I'm doing...

lassik commented 4 years ago

Ah, here's a good explanation that clears up tags in general (note to self: a tag is the part after the colon) and the misnamed default tag latest in particular: https://www.freecodecamp.org/news/an-introduction-to-docker-tags-9b5395636c2a/

lassik commented 4 years ago

So for us,

lassik commented 4 years ago

Good news (tentatively), the Chicken builds went fine. What's more, I added symbolic links to the Git repo in order to supply Docker tags that are aliases of other tags. Symlinks seem to work just as well as ordinary directories as far as Docker is concerned.

If there are no unforeseen problems with this approach, we can use track minor versions in directories like:

And then 2 can be a symlink pointing to the latest 2.x directory at any given time; and likewise for 3, etc. latest can be a symlink pointing to 3; when Guile 4 comes around, we can change it to point to 4.

lassik commented 4 years ago

We could add 2.0-alpine on the side, and 2-alpine would point to it.

This causes the interesting predicament that 2 (debian) may point to a different minor version than 2-alpine. I think that's ok, since there may be situations where a new release of a Scheme implementation builds on one distro but not on another (or we haven't yet figured out how to get it to build on the other one). In that case, to have all 2 tags point to the same minor version, we'd have to hold back on upgrading Scheme version on the distro where we would already know how to do it, until we figure out the other distro as well.

I think it's better policy to give users new minor releases ASAP, even if they don't yet work in all configurations.

There could also be rare situations where an implementation or distro makes breaking changes so that future versions of that implementation will never work on current and future versions of that distro. Then we'd have to stall that implementation's upgrades indefinitely across all distros. Not good.

lassik commented 4 years ago

What do you think about this organization for Loko?

Or should we add minor versions as well? 0.3, 0.4, etc.

lassik commented 4 years ago

We can also add minor versions to Git, but not tell Docker Hub to build them yet. It would be simpler to have a 1:1 correspondence between the directories in Git and the available Docker tags though. Especially once we can figure out how to automate Docker Hub.

lassik commented 4 years ago

The containers are getting oddly many downloads even though we aren't advertising them. Is there some automated fetching going on, or are people finding the "schemers" name from Docker's search somehow?

weinholt commented 4 years ago

There's no need to have minor versions for Loko as this point.

weinholt commented 4 years ago

The containers are getting oddly many downloads even though we aren't advertising them. Is there some automated fetching going on, or are people finding the "schemers" name from Docker's search somehow?

I blame Scheme's immense popularity. :)

lassik commented 4 years ago

We are on the cusp of world domination. Should happen any minute now.

lassik commented 4 years ago

The current arrangement (as detailed above) has been working fine in practice for weeks. Please re-open this issue or if it turns out problematic after all, or start a new issue for a specific problem.