[prebuild][selfhosted] prebuilds freeze and does not fail nor finish.

[fredrikhgrelland]

Bug description

I am running a docker build as an init-task in a self-hosted gitpod on EKS. This works fine every time when opening a new workspace. I have given the pods 16gb ram and 4 cpus.

When starting a prebuild with with the magic-link https://gitpod.example.com/#prebuild/https://github/myrepo one of the following things will happen:

• prebuild starts and says "Connecting to workspace logs..." forever
• prebuild starts, shows log output and the log stops at a random process.

For both cases i can observe:

• the prebuild pod is running and i can connect a terminal to it.
• the init-task is apparently running with supervisor init

  gitpod ~ $ ps -axf
  PID TTY      STAT   TIME COMMAND
  1136 pts/2    Ss     0:00 bash
  4287 pts/2    R+     0:00  \_ ps -axf
    1 ?        Ssl    0:00 /.supervisor/workspacekit ring0
   14 ?        Sl     0:08 /proc/self/exe ring1 --mapping-established
   28 ?        Sl     0:00  \_ supervisor init
   59 ?        Sl     0:08  |   \_ supervisor run
   78 pts/0    Ss     0:00  |       \_ /bin/bash
  402 pts/0    S+     0:00  |       |   \_ make build
  403 pts/0    S+     0:00  |       |       \_ /bin/bash -c cd aws-mwaa-local-runner && ./mwaa-local-env build-image docker image save amazon/mwaa-lo
  404 pts/0    S+     0:00  |       |           \_ /bin/bash ./mwaa-local-env build-image
  405 pts/0    Sl+    0:00  |       |               \_ docker build --rm --compress -t amazon/mwaa-local:2.2 ./docker
  417 pts/1    Ssl+   0:11  |       \_ /usr/bin/dockerd --experimental --rootless --data-root=/workspace/.docker-root --add-runtime gitpod=/usr/bin/r
  432 ?        Ssl    0:03  |           \_ containerd --config /var/run/docker/containerd/containerd.toml --log-level info
   50 ?        S      0:02  \_ /.supervisor/slirp4netns --configure --mtu=65520 --disable-host-loopback --api-socket /tmp/wskit-slirp4netns3587365082

• the logfile tail -f /workspace/.gitpod/prebuild-log-1 seems to stop at a random place in the process. (the log below came from a prebuild that never showed any logs in the web-ui

  
  {
  cd .airflow; make build
  }; exit
  gitpod /workspace/aws-advh-dataplatform-gitpod (main) $ {
  > cd .airflow; make build
  > }; exit
  cd aws-mwaa-local-runner && ./mwaa-local-env build-image
  docker image save amazon/mwaa-local:2.2 -o /workspace/airflow.tar.gz
  Sending build context to Docker daemon  23.76kB
  Step 1/28 : FROM amazonlinux
  latest: Pulling from library/amazonlinux
  20e1cc2336fb: Pull complete 
  Digest: sha256:b33b787cdb0e82495d2dc115745f68c7cd8d2585d9d83812fdc183ad39d1b753
  Status: Downloaded newer image for amazonlinux:latest
  ---> 2aa63ff55c40
  Step 2/28 : LABEL maintainer="amazon"
  ---> Running in 203f9df4ba92
  Removing intermediate container 203f9df4ba92
  ---> c004d0c0733a
  Step 3/28 : ARG AIRFLOW_VERSION=2.2.2
  ---> Running in 78149a2a1562
  Removing intermediate container 78149a2a1562
  ---> 7d43f654d576
  Step 4/28 : ARG WATCHTOWER_VERSION=1.0.6
  ---> Running in 31b0b1d870ab
  Removing intermediate container 31b0b1d870ab
  ---> c00b0a4d6209
  Step 5/28 : ARG PROVIDER_AMAZON_VERSION=2.4.0
  ---> Running in ad195792a0ad
  Removing intermediate container ad195792a0ad
  ---> 9b068285af83
  Step 6/28 : ARG AIRFLOW_USER_HOME=/usr/local/airflow
  ---> Running in 5ac5bb3631c8
  Removing intermediate container 5ac5bb3631c8
  ---> 99c2493d01ff
  Step 7/28 : ARG AIRFLOW_DEPS=""
  ---> Running in d66e0cfcc412
  Removing intermediate container d66e0cfcc412
  ---> ea7d541231ac
  Step 8/28 : ARG PYTHON_DEPS=""
  ---> Running in 2dcb6d4fa12d
  Removing intermediate container 2dcb6d4fa12d
  ---> 16a5845cedff
  Step 9/28 : ARG SYSTEM_DEPS=""
  ---> Running in f10c1be7d845
  Removing intermediate container f10c1be7d845
  ---> 9c6973f31e44
  Step 10/28 : ARG INDEX_URL=""
  ---> Running in 358c6ffd00b2
  Removing intermediate container 358c6ffd00b2
  ---> e73746a0ae16
  Step 11/28 : ENV AIRFLOW_HOME=${AIRFLOW_USER_HOME}
  ---> Running in bdfb343f7ade
  Removing intermediate container bdfb343f7ade
  ---> c312340b4605
  Step 12/28 : COPY script/bootstrap.sh /bootstrap.sh
  ---> fb6ccbf77656
  Step 13/28 : COPY script/systemlibs.sh /systemlibs.sh
  ---> 6de8ffb20c70
  Step 14/28 : COPY config/constraints.txt /constraints.txt
  ---> 284a68de72a6
  Step 15/28 : COPY config/requirements.txt /requirements.txt
  ---> 23d67fa86e0f
  Step 16/28 : COPY config/mwaa-base-providers-requirements.txt /mwaa-base-providers-requirements.txt
  ---> ee6975146786
  Step 17/28 : RUN chmod u+x /systemlibs.sh && /systemlibs.sh
  ---> Running in 3175a226e13b
  Loaded plugins: ovl, priorities
  No packages marked for update
  Loaded plugins: ovl, priorities
  No package gcc-g++ available.
  Resolving Dependencies
  --> Running transaction check
  ---> Package gcc.x86_64 0:7.3.1-13.amzn2 will be installed
  --> Processing Dependency: libgomp = 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
  --> Processing Dependency: cpp = 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
  --> Processing Dependency: libsanitizer >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
  --> Processing Dependency: libquadmath >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
  --> Processing Dependency: libmpx >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
  --> Processing Dependency: libitm >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
  --> Processing Dependency: libcilkrts >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
  --> Processing Dependency: libatomic >= 7.3.1-13.amzn2 for package: gcc-7.3.1-13.amzn2.x86_64
  --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-13.amzn2.x86_64
  --> Processing Dependency: binutils >= 2.24 for package: gcc-7.3.1-13.amzn2.x86_64
  --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-13.amzn2.x86_64
  --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-13.amzn2.x86_64
  --> Processing Dependency: libgomp.so.1()(64bit) for package: gcc-7.3.1-13.amzn2.x86_64
  ---> Package python3.x86_64 0:3.7.10-1.amzn2.0.1 will be installed
  --> Processing Dependency: python3-libs(x86-64) = 3.7.10-1.amzn2.0.1 for package: python3-3.7.10-1.amzn2.0.1.x86_64
  --> Processing Dependency: python3-setuptools for package: python3-3.7.10-1.amzn2.0.1.x86_64
  --> Processing Dependency: python3-pip for package: python3-3.7.10-1.amzn2.0.1.x86_64
  --> Processing Dependency: libpython3.7m.so.1.0()(64bit) for package: python3-3.7.10-1.amzn2.0.1.x86_64
  ---> Package python3-devel.x86_64 0:3.7.10-1.amzn2.0.1 will be installed
  --> Processing Dependency: system-rpm-config for package: python3-devel-3.7.10-1.amzn2.0.1.x86_64
  --> Processing Dependency: python3-rpm-macros for package: python3-devel-3.7.10-1.amzn2.0.1.x86_64
  --> Processing Dependency: /usr/bin/pkg-config for package: python3-devel-3.7.10-1.amzn2.0.1.x86_64
  --> Running transaction check
  ---> Package binutils.x86_64 0:2.29.1-31.amzn2 will be installed
  ---> Package cpp.x86_64 0:7.3.1-13.amzn2 will be installed
  ---> Package glibc-devel.x86_64 0:2.26-57.amzn2 will be installed
  --> Processing Dependency: glibc-headers = 2.26-57.amzn2 for package: glibc-devel-2.26-57.amzn2.x86_64
  --> Processing Dependency: glibc-headers for package: glibc-devel-2.26-57.amzn2.x86_64
  ---> Package libatomic.x86_64 0:7.3.1-13.amzn2 will be installed
  ---> Package libcilkrts.x86_64 0:7.3.1-13.amzn2 will be installed
  ---> Package libgomp.x86_64 0:7.3.1-13.amzn2 will be installed
  ---> Package libitm.x86_64 0:7.3.1-13.amzn2 will be installed
  ---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed
  ---> Package libmpx.x86_64 0:7.3.1-13.amzn2 will be installed
  ---> Package libquadmath.x86_64 0:7.3.1-13.amzn2 will be installed
  ---> Package libsanitizer.x86_64 0:7.3.1-13.amzn2 will be installed
  ---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed
  ---> Package pkgconfig.x86_64 1:0.27.1-4.amzn2.0.2 will be installed
  ---> Package python3-libs.x86_64 0:3.7.10-1.amzn2.0.1 will be installed
  --> Processing Dependency: libtirpc.so.1()(64bit) for package: python3-libs-3.7.10-1.amzn2.0.1.x86_64
  ---> Package python3-pip.noarch 0:20.2.2-1.amzn2.0.3 will be installed
  ---> Package python3-rpm-macros.noarch 0:3-60.amzn2.0.1 will be installed
  --> Processing Dependency: python-srpm-macros >= 3-38 for package: python3-rpm-macros-3-60.amzn2.0.1.noarch
  --> Processing Dependency: python-rpm-macros for package: python3-rpm-macros-3-60.amzn2.0.1.noarch
  ---> Package python3-setuptools.noarch 0:49.1.3-1.amzn2.0.2 will be installed
  ---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.10 will be installed
  --> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.10.noarch
  --> Processing Dependency: zip for package: system-rpm-config-9.1.0-76.amzn2.0.10.noarch
  --> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.10.noarch
  --> Processing Dependency: perl(Getopt::Long) for package: system-rpm-config-9.1.0-76.amzn2.0.10.noarch
  --> Processing Dependency: /usr/bin/perl for package: system-rpm-config-9.1.0-76.amzn2.0.10.noarch
  --> Running transaction check
  ---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed
  ---> Package glibc-headers.x86_64 0:2.26-57.amzn2 will be installed
  --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-57.amzn2.x86_64
  --> Processing Dependency: kernel-headers for package: glibc-headers-2.26-57.amzn2.x86_64
  ---> Package libtirpc.x86_64 0:0.2.4-0.16.amzn2 will be installed
  ---> Package perl.x86_64 4:5.16.3-299.amzn2.0.1 will be installed
  --> Processing Dependency: perl-libs = 4:5.16.3-299.amzn2.0.1 for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(Socket) >= 1.3 for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(Scalar::Util) >= 1.10 for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl-macros for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl-libs for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(threads::shared) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(threads) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(constant) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(Time::Local) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(Time::HiRes) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(Storable) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(Socket) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(Scalar::Util) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(Pod::Simple::XHTML) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(Pod::Simple::Search) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(Filter::Util::Call) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(File::Temp) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(File::Spec::Unix) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(File::Spec::Functions) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(File::Spec) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(File::Path) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(Exporter) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(Cwd) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: perl(Carp) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  --> Processing Dependency: libperl.so()(64bit) for package: 4:perl-5.16.3-299.amzn2.0.1.x86_64
  ---> Package perl-Getopt-Long.noarch 0:2.40-3.amzn2 will be installed
  --> Processing Dependency: perl(Pod::Usage) >= 1.14 for package: perl-Getopt-Long-2.40-3.amzn2.noarch
  --> Processing Dependency: perl(Text::ParseWords) for package: perl-Getopt-Long-2.40-3.amzn2.noarch
  ---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed
  ---> Package python-rpm-macros.noarch 0:3-60.amzn2.0.1 will be installed
  ---> Package python-srpm-macros.noarch 0:3-60.amzn2.0.1 will be installed
  ---> Package zip.x86_64 0:3.0-11.amzn2.0.2 will be installed
  --> Running transaction check
  ---> Package kernel-headers.x86_64 0:4.14.268-205.500.amzn2 will be installed
  ---> Package perl-Carp.noarch 0:1.26-244.amzn2 will be installed
  ---> Package perl-Exporter.noarch 0:5.68-3.amzn2 will be installed
  ---> Package perl-File-Path.noarch 0:2.09-2.amzn2 will be installed
  ---> Package perl-File-Temp.noarch 0:0.23.01-3.amzn2 will be installed
  ---> Package perl-Filter.x86_64 0:1.49-3.amzn2.0.2 will be installed
  ---> Package perl-PathTools.x86_64 0:3.40-5.amzn2.0.2 will be installed
  ---> Package perl-Pod-Simple.noarch 1:3.28-4.amzn2 will be installed
  --> Processing Dependency: perl(Pod::Escapes) >= 1.04 for package: 1:perl-Pod-Simple-3.28-4.amzn2.noarch
  --> Processing Dependency: perl(Encode) for package: 1:perl-Pod-Simple-3.28-4.amzn2.noarch
  ---> Package perl-Pod-Usage.noarch 0:1.63-3.amzn2 will be installed
  --> Processing Dependency: perl(Pod::Text) >= 3.15 for package: perl-Pod-Usage-1.63-3.amzn2.noarch
  --> Processing Dependency: perl-Pod-Perldoc for package: perl-Pod-Usage-1.63-3.amzn2.noarch
  ---> Package perl-Scalar-List-Utils.x86_64 0:1.27-248.amzn2.0.2 will be installed
  ---> Package perl-Socket.x86_64 0:2.010-4.amzn2.0.2 will be installed
  ---> Package perl-Storable.x86_64 0:2.45-3.amzn2.0.2 will be installed
  ---> Package perl-Text-ParseWords.noarch 0:3.29-4.amzn2 will be installed
  ---> Package perl-Time-HiRes.x86_64 4:1.9725-3.amzn2.0.2 will be installed
  ---> Package perl-Time-Local.noarch 0:1.2300-2.amzn2 will be installed
  ---> Package perl-constant.noarch 0:1.27-2.amzn2.0.1 will be installed
  ---> Package perl-libs.x86_64 4:5.16.3-299.amzn2.0.1 will be installed
  ---> Package perl-macros.x86_64 4:5.16.3-299.amzn2.0.1 will be installed
  ---> Package perl-threads.x86_64 0:1.87-4.amzn2.0.2 will be installed
  ---> Package perl-threads-shared.x86_64 0:1.43-6.amzn2.0.2 will be installed
  --> Running transaction check
  ---> Package perl-Encode.x86_64 0:2.51-7.amzn2.0.2 will be installed
  ---> Package perl-Pod-Escapes.noarch 1:1.04-299.amzn2.0.1 will be installed
  ---> Package perl-Pod-Perldoc.noarch 0:3.20-4.amzn2 will be installed
  --> Processing Dependency: perl(parent) for package: perl-Pod-Perldoc-3.20-4.amzn2.noarch
  --> Processing Dependency: perl(HTTP::Tiny) for package: perl-Pod-Perldoc-3.20-4.amzn2.noarch
  --> Processing Dependency: groff-base for package: perl-Pod-Perldoc-3.20-4.amzn2.noarch
  ---> Package perl-podlators.noarch 0:2.5.1-3.amzn2.0.1 will be installed
  --> Running transaction check
  ---> Package groff-base.x86_64 0:1.22.2-8.amzn2.0.2 will be installed
  ---> Package perl-HTTP-Tiny.noarch 0:0.033-3.amzn2 will be installed
  ---> Package perl-parent.noarch 1:0.225-244.amzn2.0.1 will be installed
  --> Finished Dependency Resolution

Dependencies Resolved

================================================================================ Package Arch Version Repository Size

Installing: gcc x86_64 7.3.1-13.amzn2 amzn2-core 22 M python3 x86_64 3.7.10-1.amzn2.0.1 amzn2-core 72 k python3-devel x86_64 3.7.10-1.amzn2.0.1 amzn2-core 243 k Installing for dependencies: binutils x86_64 2.29.1-31.amzn2 amzn2-core 5.8 M cpp x86_64 7.3.1-13.amzn2 amzn2-core 9.2 M dwz x86_64 0.11-3.amzn2.0.3 amzn2-core 98 k glibc-devel x86_64 2.26-57.amzn2 amzn2-core 994 k glibc-headers x86_64 2.26-57.amzn2 amzn2-core 514 k groff-base x86_64 1.22.2-8.amzn2.0.2 amzn2-core 948 k kernel-headers x86_64 4.14.268-205.500.amzn2 amzn2-core 1.2 M libatomic x86_64 7.3.1-13.amzn2 amzn2-core 46 k libcilkrts x86_64 7.3.1-13.amzn2 amzn2-core 85 k libgomp x86_64 7.3.1-13.amzn2 amzn2-core 204 k libitm x86_64 7.3.1-13.amzn2 amzn2-core 84 k libmpc x86_64 1.0.1-3.amzn2.0.2 amzn2-core 52 k libmpx x86_64 7.3.1-13.amzn2 amzn2-core 51 k libquadmath x86_64 7.3.1-13.amzn2 amzn2-core 189 k libsanitizer x86_64 7.3.1-13.amzn2 amzn2-core 641 k libtirpc x86_64 0.2.4-0.16.amzn2 amzn2-core 88 k mpfr x86_64 3.1.1-4.amzn2.0.2 amzn2-core 208 k perl x86_64 4:5.16.3-299.amzn2.0.1 amzn2-core 8.0 M perl-Carp noarch 1.26-244.amzn2 amzn2-core 19 k perl-Encode x86_64 2.51-7.amzn2.0.2 amzn2-core 1.5 M perl-Exporter noarch 5.68-3.amzn2 amzn2-core 29 k perl-File-Path noarch 2.09-2.amzn2 amzn2-core 27 k perl-File-Temp noarch 0.23.01-3.amzn2 amzn2-core 56 k perl-Filter x86_64 1.49-3.amzn2.0.2 amzn2-core 76 k perl-Getopt-Long noarch 2.40-3.amzn2 amzn2-core 56 k perl-HTTP-Tiny noarch 0.033-3.amzn2 amzn2-core 38 k perl-PathTools x86_64 3.40-5.amzn2.0.2 amzn2-core 83 k perl-Pod-Escapes noarch 1:1.04-299.amzn2.0.1 amzn2-core 52 k perl-Pod-Perldoc noarch 3.20-4.amzn2 amzn2-core 87 k perl-Pod-Simple noarch 1:3.28-4.amzn2 amzn2-core 216 k perl-Pod-Usage noarch 1.63-3.amzn2 amzn2-core 27 k perl-Scalar-List-Utils x86_64 1.27-248.amzn2.0.2 amzn2-core 36 k perl-Socket x86_64 2.010-4.amzn2.0.2 amzn2-core 49 k perl-Storable x86_64 2.45-3.amzn2.0.2 amzn2-core 78 k perl-Text-ParseWords noarch 3.29-4.amzn2 amzn2-core 14 k perl-Time-HiRes x86_64 4:1.9725-3.amzn2.0.2 amzn2-core 45 k perl-Time-Local noarch 1.2300-2.amzn2 amzn2-core 24 k perl-constant noarch 1.27-2.amzn2.0.1 amzn2-core 19 k perl-libs x86_64 4:5.16.3-299.amzn2.0.1 amzn2-core 685 k perl-macros x86_64 4:5.16.3-299.amzn2.0.1 amzn2-core 44 k perl-parent noarch 1:0.225-244.amzn2.0.1 amzn2-core 12 k perl-podlators noarch 2.5.1-3.amzn2.0.1 amzn2-core 112 k perl-srpm-macros noarch 1-8.amzn2.0.1 amzn2-core 4.7 k perl-threads x86_64 1.87-4.amzn2.0.2 amzn2-core 50 k perl-threads-shared x86_64 1.43-6.amzn2.0.2 amzn2-core 39 k pkgconfig x86_64 1:0.27.1-4.amzn2.0.2 amzn2-core 54 k python-rpm-macros noarch 3-60.amzn2.0.1 amzn2-core 14 k python-srpm-macros noarch 3-60.amzn2.0.1 amzn2-core 18 k python3-libs x86_64 3.7.10-1.amzn2.0.1 amzn2-core 9.2 M python3-pip noarch 20.2.2-1.amzn2.0.3 amzn2-core 2.0 M python3-rpm-macros noarch 3-60.amzn2.0.1 amzn2-core 12 k python3-setuptools noarch 49.1.3-1.amzn2.0.2 amzn2-core 1.1 M system-rpm-config noarch 9.1.0-76.amzn2.0.10 amzn2-core 80 k zip x86_64 3.0-11.amzn2.0.2 amzn2-core 263 k

Transaction Summary

Install 3 Packages (+54 Dependent packages)

Total download size: 67 M Installed size: 213 M Downloading packages:

Total 14 MB/s | 67 MB 00:04
Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64 1/57 Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64 2/57 Installing : cpp-7.3.1-13.amzn2.x86_64 3/57 Installing : libmpx-7.3.1-13.amzn2.x86_64 4/57 Installing : groff-base-1.22.2-8.amzn2.0.2.x86_64 5/57 Installing : perl-HTTP-Tiny-0.033-3.amzn2.noarch 6/57 Installing : 1:perl-parent-0.225-244.amzn2.0.1.noarch 7/57 Installing : perl-Pod-Perldoc-3.20-4.amzn2.noarch 8/57 Installing : perl-podlators-2.5.1-3.amzn2.0.1.noarch 9/57 Installing : 1:perl-Pod-Escapes-1.04-299.amzn2.0.1.noarch 10/57 Installing : perl-Text-ParseWords-3.29-4.amzn2.noarch 11/57 Installing : perl-Encode-2.51-7.amzn2.0.2.x86_64 12/57 Installing : perl-Pod-Usage-1.63-3.amzn2.noarch 13/57 Installing : 4:perl-macros-5.16.3-299.amzn2.0.1.x86_64 14/57 Installing : 4:perl-libs-5.16.3-299.amzn2.0.1.x86_64 15/57 Installing : perl-Exporter-5.68-3.amzn2.noarch 16/57 Installing : perl-Time-Local-1.2300-2.amzn2.noarch 17/57 Installing : perl-constant-1.27-2.amzn2.0.1.noarch 18/57 Installing : perl-Scalar-List-Utils-1.27-248.amzn2.0.2.x86_64 19/57 Installing : perl-PathTools-3.40-5.amzn2.0.2.x86_64 20/57 Installing : 4:perl-Time-HiRes-1.9725-3.amzn2.0.2.x86_64 21/57 Installing : perl-Socket-2.010-4.amzn2.0.2.x86_64 22/57 Installing : perl-Storable-2.45-3.amzn2.0.2.x86_64 23/57 Installing : perl-Carp-1.26-244.amzn2.noarch 24/57 Installing : perl-File-Temp-0.23.01-3.amzn2.noarch 25/57 Installing : perl-File-Path-2.09-2.amzn2.noarch 26/57 Installing : 1:perl-Pod-Simple-3.28-4.amzn2.noarch 27/57 Installing : perl-Getopt-Long-2.40-3.amzn2.noarch 28/57 Installing : perl-threads-shared-1.43-6.amzn2.0.2.x86_64 29/57 Installing : perl-threads-1.87-4.amzn2.0.2.x86_64 30/57 Installing : perl-Filter-1.49-3.amzn2.0.2.x86_64 31/57 Installing : 4:perl-5.16.3-299.amzn2.0.1.x86_64 32/57 Installing : zip-3.0-11.amzn2.0.2.x86_64 33/57 Installing : dwz-0.11-3.amzn2.0.3.x86_64 34/57 Installing : libatomic-7.3.1-13.amzn2.x86_64 35/57 Installing : 1:pkgconfig-0.27.1-4.amzn2.0.2.x86_64 36/57 Installing : libcilkrts-7.3.1-13.amzn2.x86_64 37/57 Installing : kernel-headers-4.14.268-205.500.amzn2.x86_64 38/57 Installing : glibc-headers-2.26-57.amzn2.x86_64 39/57 Installing : glibc-devel-2.26-57.amzn2.x86_64 40/57 Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch 41/57 Installing : system-rpm-config-9.1.0-76.amzn2.0.10.noarch 42/57 Installing : python-srpm-macros-3-60.amzn2.0.1.noarch 43/57 Installing : python-rpm-macros-3-60.amzn2.0.1.noarch 44/57 Installing : python3-rpm-macros-3-60.amzn2.0.1.noarch 45/57 Installing : libitm-7.3.1-13.amzn2.x86_64 46/57 Installing : libtirpc-0.2.4-0.16.amzn2.x86_64 47/57 Installing : python3-setuptools-49.1.3-1.amzn2.0.2.noarch 48/57 Installing : python3-libs-3.7.10-1.amzn2.0.1.x86_64 49/57 Installing : python3-3.7.10-1.amzn2.0.1.x86_64 50/57 Installing : python3-pip-20.2.2-1.amzn2.0.3.noarch 51/57 Installing : libgomp-7.3.1-13.amzn2.x86_64 52/57 Installing : binutils-2.29.1-31.amzn2.x86_64 53/57 Installing : libquadmath-7.3.1-13.amzn2.x86_64 54/57

- the last few log-lines of the pod logs are:

022-03-23T19:13:10.403583732Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-03-23T19:13:10.403375610Z] Layer sha256:ef0c7ae227b2ca9d656b7472fdf9625d0b4fb0d710446b3910689bc0b7e05926 cleaned up \r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:10Z"} 2022-03-23T19:13:11.424728801Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-03-23T19:13:11.424525877Z] Layer sha256:ef0c7ae227b2ca9d656b7472fdf9625d0b4fb0d710446b3910689bc0b7e05926 cleaned up \r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:11Z"} 2022-03-23T19:13:12.397423269Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-03-23T19:13:12.397243466Z] Layer sha256:ef0c7ae227b2ca9d656b7472fdf9625d0b4fb0d710446b3910689bc0b7e05926 cleaned up \r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:12Z"} 2022-03-23T19:13:13.401560619Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-03-23T19:13:13.401387801Z] Layer sha256:ef0c7ae227b2ca9d656b7472fdf9625d0b4fb0d710446b3910689bc0b7e05926 cleaned up \r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:13Z"} 2022-03-23T19:13:19.100783313Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"time=\"2022-03-23T19:13:19.100289175Z\" level=info msg=\"starting signal loop\" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/3175a226e13b96aca3666c1ff5da1fa89f0ff50eb375ab280dae8398876dc41f pid=774\r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:19Z"} 2022-03-23T19:13:19.273519835Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[31mERRO\u001b[0m[0000] failed to disable IPv6 forwarding for container's interface all: open /proc/sys/net/ipv6/conf/all/disable_ipv6: read-only file system \r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:19Z"} 2022-03-23T19:13:19.277003322Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[33mWARN\u001b[0m[2022-03-23T19:13:19.276839148Z] Failed to disable IPv6 on all interfaces on network namespace \"/var/run/docker/netns/0d57e6164179\": reexec to set IPv6 failed: exit status 4 \r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:19Z"} 2022-03-23T19:13:19.388745822Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"time=\"2022-03-23T19:13:19.388494215Z\" level=error msg=\"add cg to OOM monitor\" error=\"cgroups: memory cgroup not supported on this system\"\r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:13:19Z"} 2022-03-23T19:14:31.393949730Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-03-23T19:14:31.393663633Z] ignoring event \u001b[36mcontainer\u001b[0m=3175a226e13b96aca3666c1ff5da1fa89f0ff50eb375ab280dae8398876dc41f \u001b[36mmodule\u001b[0m=libcontainerd \u001b[36mnamespace\u001b[0m=moby \u001b[36mtopic\u001b[0m=/tasks/delete \u001b[36mtype\u001b[0m=\"*events.TaskDelete\"","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:14:31Z"} 2022-03-23T19:14:31.393993839Z {"label":"24db7a43-c5e3-4254-8796-045f8bcfd7fc","level":"info","message":"\r\n\u001b[36mINFO\u001b[0m[2022-03-23T19:14:31.393658438Z] shim disconnected \u001b[36mid\u001b[0m=3175a226e13b96aca3666c1ff5da1fa89f0ff50eb375ab280dae8398876dc41f\r\n\u001b[31mERRO\u001b[0m[2022-03-23T19:14:31.393791565Z] copy shim log \u001b[31merror\u001b[0m=\"read /proc/self/fd/16: file already closed\"\r\n","serviceContext":{"service":"supervisor","version":"commit-c94004de2e4ad99f5aa48d3e2f63c9afde1fadb8"},"severity":"INFO","terminalOutput":true,"time":"2022-03-23T19:14:31Z"}

### Steps to reproduce

Running a prebuild building a docker image with significant load?

### Workspace affected

_No response_

### Expected behavior

Prebuilds should fail or succed. Now stuck in limbo.

### Example repository

_No response_

### Anything else?

_No response_

[axonasif]

Hey @fredrikhgrelland, is repo public? It would help to reproduce. (maybe one of your dockerfile commands requires user input?) Prebuilds can not run longer than 1h, not sure if that's the case.

[fredrikhgrelland]

@axonasif I made a reproducible repo public here: https://github.com/fredrikhgrelland/gitpod-public-sandbox.

Case:

• [self-hosted] [without prebuild] OK https://gitpod.example.com#https://github.com/fredrikhgrelland/gitpod-public-sandbox
• [gitpod-io] [without prebuild] OK https://gitpod.io#https://github.com/fredrikhgrelland/gitpod-public-sandbox
• [self-hosted] [prebuild] FREEZE https://gitpod.example.com#prebuild/https://github.com/fredrikhgrelland/gitpod-public-sandbox
• [gitpod-io] [prebuild] OK https://gitpod.io#prebuild/https://github.com/fredrikhgrelland/gitpod-public-sandbox

[Pothulapati]

Hey @fredrikhgrelland, Thanks for the detailed issue, repro cases. Definitely helps make debugging easier. :)

So, I have a self-hosted instance running at tarun.gitpod-self-hosted.com and for me the prebuild seems to work fine and ran to completion just like gitpod-io, and you can find the logs here. I'm still finding similar error logs in the prebuild container though i.e

{"label":"4ea4451c-88f9-4d6b-aae4-b5efbff7f576","level":"info","message":"\u001b[31mERRO\u001b[0m[0000] failed to disable IPv6 forwarding for container's interface all: open /proc/sys/net/ipv6/conf/all/disable_ipv6: read-only file system \r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T06:27:23Z"}
{"label":"4ea4451c-88f9-4d6b-aae4-b5efbff7f576","level":"info","message":"\u001b[33mWARN\u001b[0m[2022-04-06T06:27:23.988344674Z] Failed to disable IPv6 on all interfaces on network namespace \"/var/run/docker/netns/05084a1153e9\": reexec to set IPv6 failed: exit status 4 \r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T06:27:23Z"}
{"label":"4ea4451c-88f9-4d6b-aae4-b5efbff7f576","level":"info","message":"time=\"2022-04-06T06:27:24.082745291Z\" level=error msg=\"add cg to OOM monitor\" error=\"open /sys/fs/cgroup/memory/cgroup.event_control: read-only file system\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T06:27:24Z"}
{"label":"4ea4451c-88f9-4d6b-aae4-b5efbff7f576","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-04-06T06:27:24.107962721Z] ignoring event                                \u001b[36mcontainer\u001b[0m=1956b3e5dbfb6d29e421c554a331e8dc341958b7b976e97ed53aefbd61560ffd \u001b[36mmodule\u001b[0m=libcontainerd \u001b[36mnamespace\u001b[0m=moby \u001b[36mtopic\u001b[0m=/tasks/delete \u001b[36mtype\u001b[0m=\"*events.TaskDelete\"","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T06:27:24Z"}
{"label":"4ea4451c-88f9-4d6b-aae4-b5efbff7f576","level":"info","message":"\r\n\u001b[36mINFO\u001b[0m[2022-04-06T06:27:24.108178124Z] shim disconnected                             \u001b[36mid\u001b[0m=1956b3e5dbfb6d29e421c554a331e8dc341958b7b976e97ed53aefbd61560ffd\r\n\u001b[31mERRO\u001b[0m[2022-04-06T06:27:24.108283005Z] copy shim log                                 \u001b[31merror\u001b[0m=\"read /proc/self/fd/16: file already closed\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T06:27:24Z"}
{"label":"4ea4451c-88f9-4d6b-aae4-b5efbff7f576","level":"info","message":"time=\"2022-04-06T06:27:25.896634250Z\" level=info msg=\"starting signal loop\" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/5d5aa37079ecb996943be96a83de875668f514a5957eee1233f6293e2da219b8 pid=1840\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T06:27:25Z"}

Debugging more on this, and will update on this once I find more!

but the prebuild should still run to completion (just like in my case) and I'm wondering why that is not the case in your environment 🤔

[fredrikhgrelland]

Yeah, it is bit of a head-scratcher. I have replicated this on a brand new cluster using the latest version of gitpod and I am still seeing it freeze-fail. It is not failing, but freezing up, and that is part of the problem.

Are you seeing these kind of logs as well: {"label":"9f1d28bc-8ffb-4f45-b286-7abbf0469cc0","level":"info","message":"time=\"2022-04-06T10:03:48.407286565Z\" level=error msg=\"add cg to OOM monitor\" error=\"open /sys/fs/cgroup/memory/cgroup.event_control: read-only file system\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T10:03:48Z"} It stops with this log-line: {"label":"9f1d28bc-8ffb-4f45-b286-7abbf0469cc0","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-04-06T10:05:00.319048376Z] ignoring event \u001b[36mcontainer\u001b[0m=c52bcf11adce366c81109ebec0dd73b3a3cb7a02fab41f38a57888bb889509a9 \u001b[36mmodule\u001b[0m=libcontainerd \u001b[36mnamespace\u001b[0m=moby \u001b[36mtopic\u001b[0m=/tasks/delete \u001b[36mtype\u001b[0m=\"*events.TaskDelete\"\r\n\u001b[31mERRO\u001b[0m[2022-04-06T10:05:00.319114925Z] copy shim log \u001b[31merror\u001b[0m=\"read /proc/self/fd/16: file already closed\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T10:05:00Z"}

[fredrikhgrelland]

I have run some more tests and I have maybe found something by looking at the prebuild-pod logs in real-time. The prebuild continues to run fine after these logs:

2022-04-06T11:22:39.097122703Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"time=\"2022-04-06T11:22:39.096690965Z\" level=info msg=\"starting signal loop\" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/cf2ce27e01a7732f763e28fb3af68072da6e3b763f45cdd7177beb437ae5a89d pid=966\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:22:39Z"}
2022-04-06T11:22:39.251712049Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[31mERRO\u001b[0m[0000] failed to disable IPv6 forwarding for container's interface all: open /proc/sys/net/ipv6/conf/all/disable_ipv6: read-only file system \r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:22:39Z"}
2022-04-06T11:22:39.254483194Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[33mWARN\u001b[0m[2022-04-06T11:22:39.254360210Z] Failed to disable IPv6 on all interfaces on network namespace \"/var/run/docker/netns/04aa21c0f7c2\": reexec to set IPv6 failed: exit status 4 \r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:22:39Z"}
2022-04-06T11:22:39.384084749Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"time=\"2022-04-06T11:22:39.383842402Z\" level=error msg=\"add cg to OOM monitor\" error=\"open /sys/fs/cgroup/memory/cgroup.event_control: read-only file system\"\r\n","serviceContext":

While logs are streaming running for ~25 seconds:

{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:22:39Z"}
2022-04-06T11:23:21.397757620Z {"alias":"1dcdd82d-03d0-4184-928d-8c1496251d31","level":"info","message":"new terminal client","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","time":"2022-04-06T11:23:21Z"}
2022-04-06T11:23:23.412642462Z {"alias":"1dcdd82d-03d0-4184-928d-8c1496251d31","level":"info","message":"new terminal client","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","time":"2022-04-06T11:23:23Z"}

Log-streaming freeze and this pops up in the logs:

2022-04-06T11:23:49.278962037Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-04-06T11:23:49.278657618Z] ignoring event                                \u001b[36mcontainer\u001b[0m=cf2ce27e01a7732f763e28fb3af68072da6e3b763f45cdd7177beb437ae5a89d \u001b[36mmodule\u001b[0m=libcontainerd \u001b[36mnamespace\u001b[0m=moby \u001b[36mtopic\u001b[0m=/tasks/delete \u001b[36mtype\u001b[0m=\"*events.TaskDelete\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:49Z"}
2022-04-06T11:23:49.279022911Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[36mINFO\u001b[0m[2022-04-06T11:23:49.278741941Z] shim disconnected                             \u001b[36mid\u001b[0m=cf2ce27e01a7732f763e28fb3af68072da6e3b763f45cdd7177beb437ae5a89d\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:49Z"}
2022-04-06T11:23:49.279031115Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[31mERRO\u001b[0m[2022-04-06T11:23:49.278834472Z] copy shim log                                 \u001b[31merror\u001b[0m=\"read /proc/self/fd/16: file already closed\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:49Z"}
2022-04-06T11:23:57.144430694Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"time=\"2022-04-06T11:23:57.144019839Z\" level=info msg=\"starting signal loop\" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/ce8dc69ef9faacc392296cd07bed61be5977a0db254d4a8494d61136a979b0b9 pid=1453\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:57Z"}
2022-04-06T11:23:57.295293487Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[31mERRO\u001b[0m[0000] failed to disable IPv6 forwarding for container's interface all: open /proc/sys/net/ipv6/conf/all/disable_ipv6: read-only file system \r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:57Z"}
2022-04-06T11:23:57.298205229Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"\u001b[33mWARN\u001b[0m[2022-04-06T11:23:57.298060250Z] Failed to disable IPv6 on all interfaces on network namespace \"/var/run/docker/netns/546576152c18\": reexec to set IPv6 failed: exit status 4 \r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:57Z"}
2022-04-06T11:23:57.431655965Z {"label":"3971cec7-cdc4-4fcb-8d85-2307a744cc94","level":"info","message":"time=\"2022-04-06T11:23:57.431393664Z\" level=error msg=\"add cg to OOM monitor\" error=\"open /sys/fs/cgroup/memory/cgroup.event_control: read-only file system\"\r\n","serviceContext":{"service":"supervisor","version":"commit-abd108b30f9e5d8dfd1b1558f19c2f86cb0830d5"},"severity":"INFO","terminalOutput":true,"time":"2022-04-06T11:23:57Z"}

The prebuild will now hang for ~1 hour and then continue to a image-build stage until starting the workspace, the prebuild clearly not done. To speed this up, I might kill the pod and repeat.

NOTE: Normally it gets in a freeze-failing state far earlier and before the web-app starts printing logs.

[fredrikhgrelland]

I have some more observations that might help further:

The webapp log-streaming as well as attaching to the prebuild pod tailing from /workspace/.gitpod/prebuild-log-1 are equal. While observing running processes inside of the docker-build of the prebuildpod using ps -axf and watching /workspace/.gitpod/prebuild-log-1 I can clearly see that the docker-build is continuing well after the logs have stopped streaming.

[corneliusludmann]

• prebuild starts and says "Connecting to workspace logs..." forever
• prebuild starts, shows log output and the log stops at a random process.

That is a UX issue. I see it from time to time as well: Prebuilds are running but the logs are not shown in the UI. Reloading the window helps for me usually. I think it's issue https://github.com/gitpod-io/gitpod/issues/8684, correct me when I'm wrong, @geropl.

The prebuild will now hang for ~1 hour and then continue to a image-build stage until starting the workspace, the prebuild clearly not done.

For prebuilds, we have a timeout of 1 hour. How long does the build usually take?

[geropl]

think it's issue https://github.com/gitpod-io/gitpod/issues/8684, correct me when I'm wrong, @geropl.

Exactly. #8684 does overlap with other issues already merged (but not deployed yet), so: we're working on this part.

@fredrikhgrelland Thanks for your very detailed analysis! :pray: As far as I understand, especially after reading this comment of yours, this is a supervisor/workspace problem:

Log-streaming freeze and this pops up in the logs:

ignoring event shim disconnected copy shim log

@kylos101 This sounds like a runtime issue. Can someone from workspace have a look at this? @sagor999 maybe? :thinking:

[kylos101]

Hi @fredrikhgrelland :wave: ,

Can you tell if you're hitting this line?

If yes, may I ask you to confirm from a regular workspace, what version of docker-compose version that you have?

I ask because about a month ago we updated the version of docker inworkspace-full, but I'm not sure what's being used with your installation. These issues were fixed in docker-compose 2.2.3, but, if you're using 2.2.2, you might be bumping into these issues. [1][2]

[fredrikhgrelland]

Hi @kylos101,

Can you tell if you're hitting this line?

Unless you are running the command steps during a prebuild, I am not. See this line

If yes, may I ask you to confirm from a regular workspace, what version of docker-compose version that you have?

I ask because about a month ago we updated the version of docker inworkspace-full, but I'm not sure what's being used with your installation. These issues were fixed in docker-compose 2.2.3, but, if you're using 2.2.2, you might be bumping into these issues. [1][2]

gitpod /workspace/gitpod-public-sandbox/.airflow (isolate-docker-compose) $ docker-compose version
Docker Compose version v2.2.3

[fredrikhgrelland]

@geropl

Is there any more steps I can take in order to debug a potential supervisor/workspace-issue? Before easter I was chasing a containerd isse that looked like it fit the bill, but am currently a bit stuck. https://github.com/containerd/containerd/issues/6402

[geropl]

@fredrikhgrelland What version of containerd are you on? :thinking:

[fredrikhgrelland]

@geropl I am running containerd github.com/containerd/containerd v1.5.8 1e5ef943eb76627a6d3b6de8cd1ef6537f393a71 This comes from the ami provided in the eks-guide https://github.com/gitpod-io/gitpod-eks-guide. My plan was to update the build with a newer binary, but I am currently unsuccessful to build an ami with the code here: https://github.com/gitpod-io/gitpod-eks-guide/tree/main/ami because of missing files. I have tried to pillage files from the running amis, but gitpod seems to fail getting up and running on a on a cluster built on my ami from source.

[kylos101]

Hey @fredrikhgrelland , are you still having trouble with prebuilds freezing?

CC: @metcalfc

[kylos101]

Hi @fredrikhgrelland , I am going to close this for now, but, if it's still a problem, let us know and we'll happily reopen. :pray:

CC: @metcalfc :eyes: