Consider splitting up the `common` feature

alexdima commented 2 years ago

We could consider deprecating the common feature (similar to some of the ones mentioned in https://github.com/devcontainers/features/issues/64) and creating multiple features out of it.

The current common feature contains (text from @Chuxel):

1. Common OS Utilities, set a locale if one not there, systemctl shim

[ ] A “common utilities” feature w/o sudo. To recap, its goal is to install:
Common command line utilities used in shell scripts (e.g., ps, lsof, wget, curl + ca-certificates zip, unzip, tar, net-tools)
Utilities frequently referenced in Linux install instructions (apt-transport-https, lsb-release, jq, gnupg2 + dirmngr)
Man pages (man-db, manpages, manpages-dev, manpages-posix manpages-posix-dev)
Libraries frequently required by native code that is not statically compiled (libc6, libgcc1, libkrb5-3, libgssapi-krb5-2, libicu[0-9][0-9], liblttng-ust[0-9], libstdc++6, zlib1g)
Give people a hint for what to do w/o systemctl since systemd is rarely in container images, but pass through if it’s there. Lots of Linux instructions reference it so it was added in time because of customer confusion.

Many of these are already in images like “node”, but they’re checked for coverage since they can be missing from minimal images like “debian”. Adding a default locale is a good idea given the number of developer tools that can bomb without it, but we could give the option to pick a locale. Right now, it’s always en_US.UTF-8. In theory we could pull out git in favor of the git feature but given how central it is to developer images and it only installs if not present, it might be worth keeping.

2. Default bash theme

[ ] Include in “common utilities” but change the default so the source code portion of it is off by default since this can be slow in massive repos. Also disabling source control is mandatory if git is removed from common utilities. Rename it to “devcontainers-default” instead of “codespaces.”

3. apt-get upgrade option

[ ] New “apt” feature which includes upgrade and the option to install packages from a list.

4. Non-root user and w/sudo setup

[ ] Add into devcontainer.json as a core option including installing, configuring sudo @chrmarti (Proposal: https://github.com/devcontainers/spec/issues/75)

5. Zsh + Oh My Zsh + Default Theme

[ ] New “ZSH” feature that includes default theme but makes it an option. Rename the theme “devcontainers-default.”

6. Set git editor to “code” if in VS Code terminal window and editor not already set

[ ] Built-in feature of Remote – Containers and Codespaces. Its presence here is a hack.

7. “code” auto-maps to code-insiders if we’re in a container created by code-insiders (since “code” won’t ever work)

[ ] Built-in feature of Remote – Containers and Codespaces. Its presence here is a hack.

8. devcontainer-info command

[ ] Separate Feature specifically used in images from https://github.com/devcontainers/images. Very useful command for debugging, but it uses a file output by the image build setup.

9. First run message

[ ] Built-in feature of Remote – Containers and Codespaces. Reuse same file location for back compat.

Its inclusion here was always a workaround. Tactically we could just make this a separate feature, but it’s always been a bit of an ugly hack.

10. common-redhat.sh, common-alpine.sh

[ ] Merge these two scripts into the “common utilities” and “zsh” features. We can detect whether apt, yum, or apk are there to figure out which scenario we are in. Most of the script is the same aside from how the packages are installed.

chrmarti commented 2 years ago

Goal

Refactor common feature into components that can be managed separately.

Parts (from devcontainers/features#67):

Common OS Utilities, set a locale if one not there, systemctl shim
Default bash theme
apt-get upgrade option
Non-root user and w/sudo setup
Zsh + Oh My Zsh + Default Theme
Set git editor to “code” if in VS Code terminal window and editor not already set
“code” auto-maps to code-insiders if we’re in a container created by code-insiders (since “code” won’t ever work)
devcontainer-info command
First run message
common-redhat.sh, common-alpine.sh

Proposal

Some of the functionality is covering the gap between base images found on Docker Hub (ubuntu, alpine, centos, etc.) which are geared towards production use and what a development setup looks like. These parts can be moved to a script for building a base image based on ubuntu. This base image can serve as the base image for all our specialized base images. (Parts: 1., 2., 3., 4.)

Zsh can be extracted to a separate feature. (Parts: 5.)

Some functionality is VS Code- or Codespaces-specific and can be moved to Remote-Containers and the CLI for Codespaces. (Parts: 6., 7., 9.)

Log messages already show /etc/os-release and other information and can be extended to show additional metadata. (Parts: 8.)

CentOS and Alpine can get separate base images. (Parts: 10.)

Notes

User creation can become part of the core functionality, but that seems unlikely to cover sudo because sudo is not included in base images like ubuntu, debian, alpine or centos and we propose that core functionality should work with any image without the need for distribution-specific code.
apt-get upgrade can be captured in the base images by frequent updates.

Chuxel commented 2 years ago

@chrmarti @alexdima

Some of the functionality is covering the gap between base images found on Docker Hub (ubuntu, alpine, centos, etc.) which are geared towards production use and what a development setup looks like. These parts can be moved to a script for building a base image based on ubuntu. This base image can serve as the base image for all our specialized base images. (Parts: 1., 2., 3., 4.)

By script do you mean "feature"? If you meant "feature", sounds good to me! However, if you literally just meant a script, I don't think that will meet the need.

Background: As I mentioned, we didn't originally had a common feature and I added it because of feedback that people wanted to reuse it with their own base images. In DockerHub alone there are a huge number of images where this applies that we don't pre-build or have templates for (e.g. I want to create a Haskell dev container, or I am using nvidia's bases because I need GPU aceleration) - let alone internal one. Companies have their own base images that will differ from ours, so they want something to add to their own definitions to make up that same delta between prod and dev container images.

samruddhikhandale commented 2 years ago

Some functionality is VS Code- or Codespaces-specific and can be moved to Remote-Containers and the CLI for Codespaces. (Parts: 6., 7., 9.)

For codespaces, instead of adding to CLI, we can probably add it to the Dockerfile which builds the codespaces image (aka universal) or even add the functionality as a local feature.

devcontainer-info command

If this is something which is only needed by the MCR images and won't be usable to the community, we can add it as a local-feature to all the image definitions (eg - https://github.com/devcontainers/images/tree/main/src/codespaces/local-features). That way we don't have to add it to the published features set

chrmarti commented 2 years ago

By script do you mean "feature"? If you meant "feature", sounds good to me! However, if you literally just meant a script, I don't think that will meet the need.

Background: As I mentioned, we didn't originally had a common feature and I added it because of feedback that people wanted to reuse it with their own base images. In DockerHub alone there are a huge number of images where this applies that we don't pre-build or have templates for (e.g. I want to create a Haskell dev container, or I am using nvidia's bases because I need GPU aceleration) - let alone internal one. Companies have their own base images that will differ from ours, so they want something to add to their own definitions to make up that same delta between prod and dev container images.

We mean script because we are thinking of it as being part of the base image (not an add-on feature). We also propose to first build a base image from Ubuntu using the common script and then build our tech-stack specific images from that common base image.

For users bringing their own base image the common script would be a convenience (I assume they usually know how to write their own Dockerfile).

Chuxel commented 2 years ago

We mean script because we are thinking of it as being part of the base image (not an add-on feature). We also propose to first build a base image from Ubuntu using the common script and then build our tech-stack specific images from that common base image.

For users bringing their own base image the common script would be a convenience (I assume they usually know how to write their own Dockerfile).

@chrmarti @alexdima Yes - but all features are a convenience. But keep in mind there's lots of base images out there without having to write a Dockerfile. One of the problems with scripts has been maintenance and up-keep. Features are an improved version of curl'ing a script and therefore the same logic applies here. Originally that was the directions for all scripts that turned into features... bash -c "$(curl -sSL <url>)" But that's not particularly secure, so there was then an automated hash generation and logic to point to a sub version rather than main - and it quickly got out of hand.

I think the thing is that I disagree with the assertion that this is limited to only our base images. Other base images exist that people will start from as I outlined above. With that in mind, following a different pattern in this one case does not make sense to me given is usefulness.This is a feature designed to bring a production image to a dev container image as you said -- which is broadly useful. Every definition in vscode-dev-containers except for two uses it including 3rd party contributed ones. We also have concrete evidence of customer demand. As we move all definitions to use features, I want developers to be able to do the following and have a fully functional dev container environment:

{
    "image": "haskell",
    "features": {
        "devcontainers/features/common-utils": "latest"
    }
}

or

{
    "image": "ghcr.io/my-company/some-image-my-opts-team-created-that-I-didnt-create:latest",
    "features": {
        "devcontainers/features/common-utils": "latest"
    }
}

jkeech commented 2 years ago

I agree with @Chuxel -- this seems broadly useful and re-usable outside of our base images. As a general principal, I think all of the scripts/logic that we use for the base images should be exposed as re-usable features for the community. Features are the building block that makes sharing and maintaining that logic easier.

Basically there's no real downside that I can see in wrapping the logic in a feature. The outcome in the official base images is the same. But the upside is that it's much easier for someone to pick up and include the feature in their own custom image that doesn't derive from one of our official base images. That seems like a positive to me.

Since the implementations are distro-specific, I could see us having a debian-common-utils feature and an alpine-common-utils feature, etc.

craiglpeters commented 2 years ago

See a community workaround for the first-run-notice.txt using DOCKERFILE https://github.com/DenverCoder1/github-readme-streak-stats/commit/9237db556271d3199972528d77873ac7b93636d5

eitsupi commented 1 year ago

New “apt” feature which includes upgrade and the option to install packages from a list.

Hello. I think this apt Feature is in demand as it is very versatile, do you have any plans to add it? (I'm thinking it would be useful to have it regardless of the division of the common feature) Of course I could implement this in my own repository, but since it is generic I think it would be good to have it here.

eitsupi commented 1 year ago

Hi, I published apt-packages Feature to install apt packages. https://github.com/rocker-org/devcontainer-features/tree/main/src/apt-packages

Something better than (perhaps supports more than just apt?) may become available here in the future, but for now we can use it to install any apt package without a Dockerfile.

Roemer commented 7 months ago

I think splitting it would be a good idea. Often only the user and bash or zsh configuration is needed and not all the other stuff (eg. git is also installed). All in all, the layer (according to wagoodman/dive) is a whopping 185 MB. This is especially useful if one also plans on using the dev-container image not just for developers but also for the pipeline. Other features are also not quiet optimized, eg. the git feature needs almost 500 MB in a layer when compiling from source, as a lot of files needed during compilation are not cleared.

devcontainers / features