Add devcontainer.json for development with VS Code in a Docker container

mkoeppe commented 2 years ago

https://code.visualstudio.com/docs/remote/containers

We add two new sections in the developer's guide: https://47f8b984ef964a5aa34147393fbdc32e0dde88ad--sagemath-tobias.netlify.app/developer/portability_testing.html#using-our-pre-built-docker-images-published-on-ghcr-io

We also set up devcontainer configurations for the CoCalc and computop/sage Docker images, as well as downstream distribution packaging of Sage.

https://github.com/3-manifolds/sagedocker/issues/2

Tested devcontainer.json of

portability-ubuntu-jammy-standard: builds well; runs well
develop-docker-computop: builds well; runs well
downstream-docker-cocalc: builds well; runs well (except machines with the issue #32434)
downstream-docker-computop: builds well; runs well
downstream-archlinux-latest: builds well; runs well
downstream-conda-forge-latest: builds well; runs well

Follow-ups:

34363 Installation guide: Document installation on Windows via devcontainer
34286 Cookiecutter for Sage user projects with devcontainer
34403 GitHub Codespaces - may also be able to test it by starting a codespace at https:https://user-images.githubusercontent.com/5037600/216875765-6697bbf8-70f9-4acc-868e-9309db67be92.jpg/new?hide_repo_select=true&ref=public%2Fbuild%2Fdevcontainer&repo=302607352&machine=standardLinux32gb&devcontainer_path=.devcontainer%2Fdevcontainer.json
downstream-* for more distributions (generate with tox from the info in tox.ini, use_sagemath` package)
develop-conda-forge-src-environment-dev (https://doc.sagemath.org/html/en/installation/conda.html#using-conda-to-provide-all-dependencies-for-the-sage-library-experimental, similar to our gitpod configuration)
Restore portability-debian-buster-i386-standard and fix it
Restore develop-docker-cocalc (was: builds well (after increasing disk image space on the Docker daemon); fails to run)
Restore - portability-centos-7-devtoolset-gcc_11-standard (was: builds well; fails to run (exactly the same on WSL))
Restore downstream-docker-sagemath (after #34242)
add github workflows that verify that sage is at least compiling in these containers. That should be relatively straightforward using https://github.com/devcontainers/ci
improve portability-Dockerfile as explained in comment:299

Depends on #33873 Depends on #34352

CC: @tobiasdiez @dimpase @williamstein @culler @saraedum @kwankyu

Component: user interface

Author: Tobias Diez, Matthias Koeppe, Kwankyu Lee

Branch/Commit: 4affef2

Reviewer: Kwankyu Lee, Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/33671

tobiasdiez commented 2 years ago

Reviewer: Tobias Diez (-1), ...

tobiasdiez commented 2 years ago

comment:36

You created a reusable infrastructure for something that is needed in one place and thus doesn't have to be reused. For this abstraction you are paying the usual costs that come with an additional level of indirection in terms of simplicity, discoverability, maintainability etc. I think its fair to criticize this as part of a review. So -1 from my side for the current solution to install gpgconf into the devcontainer.

mkoeppe commented 2 years ago

comment:37

There is always a trade off. I made the abstraction in full awareness of the indirection costs, and it has served the project well. Keeping the system package data in a central place goes a long way for maintainability and discoverability.

mkoeppe commented 2 years ago

comment:38

Replying to @tobiasdiez:

You created a reusable infrastructure for something that is needed in one place and thus doesn't have to be reused.

The package scripts are used for informing the user, for provisioning environments, and for generating documentation. Multiple uses, guaranteeing that things don't fall out of sync.

tobiasdiez commented 2 years ago

comment:39

Replying to @mkoeppe:

Replying to @tobiasdiez:

You created a reusable infrastructure for something that is needed in one place and thus doesn't have to be reused.

The package scripts are used for informing the user, for provisioning environments, and for generating documentation. Multiple uses, guaranteeing that things don't fall out of sync.

I don't mean the general package infrastructure, but the _develop package and in particular that gpgconf is installed there. Your advantages don't apply there: its not used to inform the user, its only used for the devcontainer, it is not displayed in the documentation, has only a single use and thus cannot fall out of sync. If in the future there might be the need to reuse it somewhere else then given the usecase one could decide about the right form of abstraction (e.g. bundling with other "devtools" or as an ordinary optional package or ...).

mkoeppe commented 2 years ago

comment:40

Replying to @tobiasdiez:

it is not displayed in the documentation

Yes it is, it's automatic.

mkoeppe commented 2 years ago

comment:41

Replying to @tobiasdiez:

If in the future [...] decide about the right form of abstraction (e.g. bundling with other "devtools" or as an ordinary optional package or ...).

It's not like these packages are mysterious VS Code-specific requirements. gpg is a typical dev tool that people use for signing commits, and ssh is also what trac users need to push.

Actually, from my use of repology for finding the gpgconf it seems that Debian is the only distro family that has it as a separate package. So one could consider to also package the full gpg in the case of Debian. That's a possible refinement.

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 44493f0 to 47f8b98

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`9e674ca`	`.devcontainer/Dockerfile: Add argument DOCKER_TAG`
`d59144c`	`src/doc/en/developer/portability_testing.rst: Add section 'Using our pre-built Docker images published on ghcr.io'`
`47f8b98`	`src/doc/en/developer/portability_testing.rst: Add section 'Developing with our pre-built Docker images as devcontainers in VS Code'`

mkoeppe commented 2 years ago

comment:43

Given that we ask users to edit the devcontainer.json file, perhaps we should only check in a template to avoid merge conflicts?

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -1,3 +1,7 @@
 https://code.visualstudio.com/docs/remote/containers

-Can be tested by starting a codespace at https://github.com/codespaces/new?hide_repo_select=true&ref=public%2Fbuild%2Fdevcontainer&repo=302607352&machine=standardLinux32gb&devcontainer_path=.devcontainer%2Fdevcontainer.json (might not be possible for everyone as codespaces is still in the test phase).
+We add two new sections in the developer's guide:
+https://47f8b984ef964a5aa34147393fbdc32e0dde88ad--sagemath-tobias.netlify.app/developer/portability_testing.html#using-our-pre-built-docker-images-published-on-ghcr-io
+
+Users who have access to GitHub Codespaces may also be able to test it by starting a codespace at https://github.com/codespaces/new?hide_repo_select=true&ref=public%2Fbuild%2Fdevcontainer&repo=302607352&machine=standardLinux32gb&devcontainer_path=.devcontainer%2Fdevcontainer.json
+

mkoeppe commented 2 years ago

comment:45

Perhaps it would also make sense to include another sample configuration for using sagemathinc/cocalc as a devcontainer?

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 47f8b98 to a7a8d66

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`fc19e7a`	`devcontainer-ubuntu-jammy-standard.json: Renamed from devcontainer.json`
`f3cbc53`	`.devcontainer: Factor out script post_create.sh`
`a7a8d66`	`.devcontainer/devcontainer-cocalc.json: New`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

0d573da src/doc/en/developer/portability_testing.rst: Copy or symlink the devcontainer.json sample file

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from a7a8d66 to 0d573da

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`aea8531`	`build/pkgs/_develop/dependencies: Add _bootstrap`
`be191a1`	`.devcontainer/devcontainer: Use MAKE='make -j12'`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 0d573da to be191a1

mkoeppe commented 2 years ago

comment:49

This works, but getting /usr/local/sage/local/lib/libgmp.a: error adding symbols: archive has no index; run ranlib to add one on cocalc

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from be191a1 to c23008b

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`b863c5c`	`.devcontainer/post_create.sh: Handle option --sudo`
`c23008b`	`.devcontainer/devcontainer-computop-sage.json: New`

mkoeppe commented 2 years ago

comment:51

computop/sage has SAGE_ROOT set in the global environment. So this needs #33786.

mkoeppe commented 2 years ago

Changed dependencies from #33851 to #33851, #33786

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from c23008b to 9cd17f9

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`60bcbec`	`README.md, src/doc/en/installation/source.rst: Update multi-user install`
`2116f9f`	`README.md: Update final step 'symlink sage', add jupyter kernel step`
`5c68335`	`Merge #33787`
`252372d`	`SAGE_ROOT/sage: Unconditionally determine SAGE_ROOT from $0; no longer invite to edit this file`
`01caa38`	`SAGE_ROOT/sage: Actually unconditionally determine SAGE_ROOT from $0`
`b9c9764`	`Merge #33786`
`9cd17f9`	`.devcontainer/devcontainer-computop-sage.json: Stop it from injecting old SAGE_ROOT into every shell`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`530eaac`	`src/bin/sage-env: Key SAGE_ENV_SOURCED also to SAGE_SRC because the PATH setting depends on it`
`c25bd9d`	`build/make/Makefile.in: Put SAGE_SRC/bin in front of SAGE_VENV/bin so that sage-env-config is sourced from SAGE_SRC`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 9cd17f9 to c25bd9d

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

9195e15 .devcontainer/devcontainer-computop-sage.json: More workarounds

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from c25bd9d to 9195e15

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -5,3 +5,6 @@

 Users who have access to GitHub Codespaces may also be able to test it by starting a codespace at https://github.com/codespaces/new?hide_repo_select=true&ref=public%2Fbuild%2Fdevcontainer&repo=302607352&machine=standardLinux32gb&devcontainer_path=.devcontainer%2Fdevcontainer.json

+We also set up devcontainer configurations for the CoCalc and computop/sage Docker images.
+- https://github.com/3-manifolds/sagedocker/issues/2
+

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`2edf926`	`.devcontainer/post_create.sh: First update package lists, also install _prereq`
`c173378`	`.devcontainer/devcontainer-sagemath-sagemath.json: New`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 9195e15 to c173378

mkoeppe commented 2 years ago

Description changed:

--- 
+++ 
@@ -5,6 +5,6 @@

 Users who have access to GitHub Codespaces may also be able to test it by starting a codespace at https://github.com/codespaces/new?hide_repo_select=true&ref=public%2Fbuild%2Fdevcontainer&repo=302607352&machine=standardLinux32gb&devcontainer_path=.devcontainer%2Fdevcontainer.json

-We also set up devcontainer configurations for the CoCalc and computop/sage Docker images.
+We also set up devcontainer configurations for the CoCalc,  computop/sage, and sagemath/sagemath Docker images.
 - https://github.com/3-manifolds/sagedocker/issues/2

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

`b1dd79d`	`.devcontainer/devcontainer-archlinux-latest-downstream.json: New`
`3c7f6bd`	`build/pkgs/_develop/dependencies: Add python3, tox`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from c173378 to 3c7f6bd

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 3c7f6bd to 16fd4af

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

16fd4af .devcontainer/devcontainer-archlinux-latest-downstream.json: Install pip, notebook

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

3c4929e .devcontainer/devcontainer-conda-forge-latest-downstream.json: New

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 16fd4af to 3c4929e

mkoeppe commented 2 years ago

Changed dependencies from #33851, #33786 to #33851, #33786, #33870

mkoeppe commented 2 years ago

Work Issues: merge #33870, use _sagemath script package

mkoeppe commented 2 years ago

Changed dependencies from #33851, #33786, #33870 to #33851, #33786, #33870, #33873

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`a2557d4`	`build/pkgs/_sagemath/distros/debian.txt: Move debian info here from src/doc/en/installation/linux.rst`
`3dd0d18`	`build/pkgs/_sagemath/type: New`
`f45f500`	`Merge #33870`
`ab5a4f1`	`build/bin/sage-get-system-packages: If system=auto, use sage-guess-package-system`
`798f4d3`	`build/bin/sage-print-system-package-command: If system=auto, use sage-guess-package-system`
`ef249cc`	`build/bin/sage-print-system-package-command: New option --spkg`
`d75da7e`	`tox.ini: Simplify using new options of sage-print-system-package-command`
`780e747`	`Merge #33873`
`d27f5e2`	`.devcontainer/post_create.sh: Simplify using new options of sage-print-system-package-command`
`df2216b`	`.devcontainer/devcontainer-archlinux-latest-downstream.json: Use spkg _sagemath`

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from 3c4929e to df2216b

mkoeppe commented 2 years ago

Changed work issues from merge #33870, use _sagemath script package to none

tobiasdiez commented 2 years ago

comment:67

Codespaces are not working as the devcontainer file is not recognized. According to https://github.blog/2022-04-20-codespaces-multi-repository-monorepo-scenarios/:

Codespaces now supports multiple devcontainer.json files inside of your .devcontainer directory, as long as they follow the pattern of .devcontainer/${DIR}/devcontainer.json

mkoeppe commented 2 years ago

comment:68

Interesting

mkoeppe commented 2 years ago

comment:69

I was looking for this feature in VS Code...

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Changed commit from df2216b to bcf4302

7ed8c4ca-6d56-4ae9-953a-41e42b4ed313 commented 2 years ago

Branch pushed to git repo; I updated commit sha1. New commits:

bcf4302 .devcontainer: Move devcontainer.json files to separate directories for Codespaces

sagemath / sage

Add devcontainer.json for development with VS Code in a Docker container #33671

34363 Installation guide: Document installation on Windows via devcontainer

34286 Cookiecutter for Sage user projects with devcontainer