Fix broken gitpod

[mkoeppe]

as reported in https://groups.google.com/g/sage-devel/c/JIiytzMsLCU

Here we

• improve error handling in the gitpod initialization scripts that eliminates the guesswork when something goes wrong
• fix a mistake in the system package information for conda

CC: @soehms @dimpase @tobiasdiez

Component: build

Author: Matthias Koeppe

Branch/Commit: 8b1063c

Reviewer: Tobias Diez

Issue created by migration from https://trac.sagemath.org/ticket/34623

[mkoeppe]

comment:1

The conda environment creation fails (mamba env create ..., see .gitpod.yml "init" task)

(base) gitpod@sagemath-sagetracmirror-6rgcwwh5cer:/workspace/sagetrac-mirror$ mamba env create --file src/environment-dev.yml --prefix venv
conda-forge/noarch                                   9.4MB @   3.0MB/s  3.2s
conda-forge/linux-64                                25.7MB @   4.4MB/s  6.0s

Looking for: ['compilers', 'make', 'm4', 'perl', 'python', 'tar', 'bc', 'pkg-config', 'appdirs', 'arb', 'boost-cpp', 'brial', 'bzip2', 'cddlib', 'cliquer', 'cmake', 'curl', 'distlib', 'ecl', 'eclib', 'ecm', 'fflas-ffpack', 'filelock', 'libflint', 'flintqs', 'fplll', 'freetype', 'bdw-gc', 'gengetopt', 'gf2x', 'gfan', 'fortran-compiler', 'giac', 'givaro', 'glpk', 'gmp', 'gsl', 'iml', 'lcalc', 'libatomic_ops', 'libbraiding', 'libffi', 'libgd', 'libhomfly', 'xz', 'libpng', 'linbox', 'lrcalc', 'm4ri', 'm4rie', 'mpc', 'mpfi', 'mpfr', 'nauty', 'ncurses', 'ninja', 'ntl', 'openblas', 'blas=2[build=openblas]', 'openssl', 'palp', 'pari=[build=*_pthread]', 'pari-elldata', 'pari-galdata', 'pari-galpol', 'pari-seadata', 'pari-galdata', 'pari-seadata-small', 'patch', 'pcre', 'pkg-config', 'planarity', 'ppl', 'primecount', 'primesieve', 'qhull', 'r', 'r-essentials', 'readline', 'rw', 'singular', 'sqlite', 'suitesparse', 'symmetrica', 'sympow', 'tachyon', 'toml', 'tox', 'virtualenv', 'xz', 'zeromq', 'zlib', 'zn_poly', 'autoconf', 'automake', 'libtool', 'alabaster', 'appnope', 'argon2-cffi', 'asttokens', 'attrs', 'babel', 'backcall', 'backports.zoneinfo', 'beautifulsoup4', 'beniget', 'bleach', 'certifi', 'cffi', 'charset-normalizer', 'sagemath-db-combinatorial-designs', 'sagemath-db-conway-polynomials', 'cppy', 'cvxopt', 'cycler', 'cypari2', 'cysignals', 'cython', 'python-dateutil', 'decorator', 'defusedxml', 'deprecation', 'docutils', 'editables', 'sagemath-db-elliptic-curves', 'entrypoints', 'executing', 'python-fastjsonschema', 'flit-core', 'fonttools', 'fpylll', 'furo', 'gap-defaults', 'gast', 'gmpy2', 'sagemath-db-graphs', 'hatchling', 'html5lib', 'idna', 'imagesize', 'importlib_metadata', 'importlib-resources', 'ipykernel', 'ipython', 'ipython_genutils', 'ipywidgets', 'jedi', 'jinja2', 'jmol', 'jsonschema', 'jupyter_client', 'jupyter_core', 'jupyter-jsmol', 'jupyter-packaging', 'jupyter_sphinx', 'jupyterlab_pygments', 'kiwisolver', 'python-lrcalc', 'markupsafe', 'mathjax', "matplotlib[version='>=3.5.1']", 'matplotlib-inline', 'maxima', 'memory-allocator', 'mistune', 'mpmath', 'nbclient', 'nbconvert', 'nbformat', 'nest-asyncio', 'networkx', 'notebook', 'numpy', 'packaging', 'pandocfilters', 'parso', 'pathspec', 'pexpect', 'pickleshare', 'pillow', 'pip', 'pkgconfig', 'platformdirs', 'pluggy', 'ply', 'poetry-core', 'sagemath-db-polytopes', 'pplpy', 'primecountpy', 'prometheus_client', 'prompt_toolkit', 'ptyprocess', 'pure_eval', 'py', 'pybind11', 'pycparser', 'pygments', 'pyparsing', 'pyrsistent', 'pythran', 'pytz', 'pytz-deprecation-shim', 'pyzmq', 'requests', 'rpy2', 'sagetex', 'scipy', 'send2trash', "setuptools[version='<64']", 'setuptools_scm', 'setuptools-scm-git-archive', 'simplegeneric', 'six', 'snowballstemmer', 'soupsieve', 'sphinx', 'sphinxcontrib-applehelp', 'sphinxcontrib-devhelp', 'sphinxcontrib-htmlhelp', 'sphinxcontrib-jsmath', 'sphinxcontrib-qthelp', 'sphinxcontrib-serializinghtml', 'sphinxcontrib-websupport', 'stack_data', 'sympy', 'terminado', 'threejs-sage=122', 'tinycss2', 'tomli', 'tomlkit', 'tornado', 'traitlets', 'typing_extensions', 'tzdata', 'tzlocal', 'urllib3', 'vcversioner', 'wcwidth', 'webencodings', 'wheel', 'widgetsnbextension', 'zipp', 'openssh', 'pycodestyle', 'esbonio', 'git']

Encountered problems while solving:
  - nothing provides requested setuptools-scm-git-archive

[mkoeppe]

Branch: u/mkoeppe/fix_broken_gitpod

[mkoeppe]

New commits:

cc24ab1

.gitpod.yml: Stop when errors occur in the init/command scripts

[mkoeppe]

Commit: cc24ab1

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

ad3a0a7

.gitpod.yml: Stop when errors occur in the init/command scripts

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from cc24ab1 to ad3a0a7

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from ad3a0a7 to 5e73805

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

5e73805

.gitpod.yml: Stop when errors occur in the init/command scripts

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 5e73805 to 40e6f51

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

40e6f51

.gitpod.yml: Stop when errors occur in the init/command scripts

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 40e6f51 to 2d8fdae

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

2d8fdae

.gitpod.yml: Stop when errors occur in the init/command scripts

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 2d8fdae to d1fcd37

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

d1fcd37

build/pkgs/setuptools_scm_git_archive/distros/conda.txt: Remove - not available in the conda-forge channel

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1 +1,6 @@
 as reported in https://groups.google.com/g/sage-devel/c/JIiytzMsLCU
+
+Here we 
+- improve error handling in the gitpod initialization scripts that eliminates the guesswork when something goes wrong 
+- fix a mistake in the system package information for conda
+

[mkoeppe]

Author: Matthias Koeppe

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from d1fcd37 to 9f055b5

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

9f055b5

.gitpod-command.sh: Fix up

[tobiasdiez]

comment:12

Thanks for working on this.

that eliminates the guesswork when something goes wrong

Not sure what guesswork you mean, but the full logs can be found at https://gitpod.io/t/sagemath/sagetrac-mirror-1/prebuilds or directly in the browser if you trigger a prebuild using the magical url.

The logs (https://gitpod.io/t/sagemath/sagetrac-mirror-1/41eac415-6e5a-41a3-9175-ac2fc463d659) for this branch indicate the following error

The tasks executed in the prebuild returned a non-zero exit code. headless task failed: exit status 1

but I couldn't find which command triggered it.

Another remark: Not sure if we really want to blow up the root folder by the additional gitpod scripts. Given that they should strongly simplify once we move to github (no extra trac nor ssh treatment will be necessary), I would propose to keep them inline.

[mkoeppe]

comment:14

Replying to Tobias Diez:

Thanks for working on this.

that eliminates the guesswork when something goes wrong

Not sure what guesswork you mean

The one in ​https://groups.google.com/g/sage-devel/c/JIiytzMsLCU trying to diagnose based on error messages of commands that ran even though an earlier command failed.

[mkoeppe]

comment:15

Replying to Tobias Diez:

I would propose to keep them inline.

I moved it to separate scripts so that there is proper error handling. Whatever the gitpod folks do with the scripts, normal error handling is not working.

[mkoeppe]

comment:16

Replying to Tobias Diez:

Given that they should strongly simplify once we move to github (no extra trac nor ssh treatment will be necessary)

Good point. I'll move that part to a separate script and see if I can keep the other things inline.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

9b955c0

.gitpod-setup-trac-remote.sh: New, move other scripts back into .gitpod.yml, chained with &&

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 9f055b5 to 9b955c0

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

400b680

.gitpod-setup-trac-remote.sh: New, move other scripts back into .gitpod.yml, chained with &&

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 9b955c0 to 400b680

[mkoeppe]

comment:19

prebuild is still running but looking good

[mkoeppe]

comment:20

Replying to Tobias Diez:

the full logs can be found at https://gitpod.io/t/sagemath/sagetrac-mirror-1/prebuilds or directly in the browser if you trigger a prebuild using the magical url.

Sure, but a problem is that when starting the gitpod, the terminal did not have enough scrollback, so one could not see the first error.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 400b680 to 8b1063c

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

8b1063c

.gitpod.yml: Fix up comment placement

[tobiasdiez]

comment:22

Thanks for the quick follow-up. Code looks good to me, gitpod prebuild and actual start are working now again. So good to go.

Could you please open an issue at the gitpod repo with your error handling problem. That should be fixed upstream.

[tobiasdiez]

Reviewer: Tobias Diez

[mkoeppe]

comment:23

Thanks!

[mkoeppe]

comment:24

I've opened https://github.com/gitpod-io/gitpod/issues/13508

[soehms]

comment:25

Thank you for fixing this immediately!

Just to understand why it worked in former versions: The error has been triggered by the presence of build/pkgs/setuptools_scm_git_archive/distros/conda.txt which has been introduced recently in #33613, right?

[mkoeppe]

comment:26

Yes, that's right. It's a mistake I made in that ticket - that conda package does exist, but only in the default channel, not in the conda-forge channel.

[soehms]

comment:27

Replying to Matthias Köppe:

Yes, that's right. It's a mistake I made in that ticket - that conda package does exist, but only in the default channel, not in the conda-forge channel.

I see. Thanks!

[vbraun]

Changed branch from u/mkoeppe/fix_broken_gitpod to 8b1063c