CI fails on azure 32 bit linux Linux_i386: sudo apt-fast install fails

timotheecour commented 3 years ago

I'm seeing: nim-lang.Nim (packages Linux_i386) Failing after 58s — packages Linux_i386 failed in several unrelated PRs (eg https://github.com/nim-lang/Nim/pull/17257#issuecomment-796247652)

eg https://dev.azure.com/nim-lang/255dfe86-e590-40bb-a8a2-3c0295ebdeb1/_apis/build/builds/14075/logs/38

2021-03-10T21:10:24.0269335Z CC: stdlib_assertions.nim
2021-03-10T21:10:24.0288424Z CC: stdlib_dollars.nim
2021-03-10T21:10:24.1410956Z Error: execution of an external compiler program 'gcc -c  -w -fmax-errors=3  -I/home/vsts/work/1/s/lib -I/home/vsts/work/1/s -o /home/vsts/.cache/nim/koch_d/stdlib_assertions.nim.c.o /home/vsts/.cache/nim/koch_d/stdlib_assertions.nim.c' failed with exit code: 1
2021-03-10T21:10:24.1412125Z 
2021-03-10T21:10:24.1412666Z In file included from /usr/include/limits.h:25:0,
2021-03-10T21:10:24.1413573Z                  from /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed/limits.h:168,
2021-03-10T21:10:24.1414497Z                  from /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed/syslimits.h:7,
2021-03-10T21:10:24.1415407Z                  from /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed/limits.h:34,
2021-03-10T21:10:24.1416047Z                  from /home/vsts/work/1/s/lib/nimbase.h:268,
2021-03-10T21:10:24.1416680Z                  from /home/vsts/.cache/nim/koch_d/stdlib_assertions.nim.c:9:
2021-03-10T21:10:24.1417570Z /usr/include/features.h:367:25: fatal error: sys/cdefs.h: No such file or directory
2021-03-10T21:10:24.1418154Z compilation terminated.
2021-03-10T21:10:24.1472789Z ##[error]Bash exited with code '1'.
2021-03-10T21:10:24.1576588Z ##[section]Finishing: Build koch

(refs https://github.com/nim-lang/Nim/issues/3123#issuecomment-122125096) refs:

sudo apt-get install g++-multilib https://stackoverflow.com/questions/51765069/fatal-error-sys-cdefs-h-no-such-file-or-directory-while-compiling-openssl-1-1
libc6-dev-i386 https://itectec.com/ubuntu/ubuntu-fatal-error-sys-cdefs-h-no-such-file-or-directory/

timotheecour commented 3 years ago

https://dev.azure.com/nim-lang/Nim/_build/results?buildId=14079&view=logs&jobId=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&j=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&t=38200048-f2b3-5d1c-03e3-3014d58a3656 shows the 1st place where there's a problem:

instead of:

Starting: Install dependencies (i386 Linux)
==============================================================================
Task         : Bash
Description  : Run a Bash script on macOS, Linux, or Windows
Version      : 3.182.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/372abbb7-b8e2-4be2-a494-3b385cd3c98e.sh
Package: libllvm6.0
Pin: origin "azure.archive.ubuntu.com"
Pin-Priority: 1001
[apt-fast 08:23:46]
[apt-fast 08:23:46]Working... this may take a while.
[apt-fast 08:23:46]

03/10 08:23:50 [NOTICE] Verification finished successfully. file=/var/cache/apt/apt-fast/libgpg-error0_1.21-2ubuntu1_i386.deb

03/10 08:23:50 [NOTICE] Download complete: /var/cache/apt/apt-fast/libgpg-error0_1.21-2ubuntu1_i386.deb

as in https://dev.azure.com/nim-lang/Nim/_build/results?buildId=14063&view=logs&jobId=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&j=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&t=38200048-f2b3-5d1c-03e3-3014d58a3656

we get:

https://dev.azure.com/nim-lang/Nim/_build/results?buildId=14079&view=logs&jobId=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&j=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&t=38200048-f2b3-5d1c-03e3-3014d58a3656

Starting: Install dependencies (i386 Linux)
==============================================================================
Task         : Bash
Description  : Run a Bash script on macOS, Linux, or Windows
Version      : 3.182.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/38167460-f4b0-4c90-96fb-020d4d6a6b8f.sh
Package: libllvm6.0
Pin: origin "azure.archive.ubuntu.com"
Pin-Priority: 1001
[apt-fast 23:14:09]
[apt-fast 23:14:09]Working... this may take a while.
E: Unable to correct problems, you have held broken packages.
[apt-fast 23:14:09]Package manager quit with exit code.
Finishing: Install dependencies (i386 Linux)

note that we should abort right here after E: Unable to correct problems, you have held broken packages. instead of continuing

alaviss commented 3 years ago

note that we should abort right here after E: Unable to correct problems, you have held broken packages. instead of continuing

Hmm, apt-fast should have exited right here...

In bash without set -e, the exit code is based on the last command run, and the last command was apt-fast. So I guess apt-fast didn't error out?

timotheecour commented 3 years ago

In bash without set -e, the exit code is based on the last command run, and the last command was apt-fast. So I guess apt-fast didn't error out?

I think it did but not using set -e is a bug as it keeps going instead of error-ing immediately. Even if the exit code is based on last command, you'd get a false success with:

cmd that fails
cmd that succeeds

as I've observed in https://github.com/nim-lang/Nim/pull/17326, we need to add set -e in the bash section to exit on 1st error; I will do that in upcoming PR to fix this

I have 1 PR that fixes CI on i386 here: https://github.com/nim-lang/Nim/pull/17327 by upgrading to ubuntu-18.04' on i386
I'm curious whether it's possible to keep ubuntu-16.04' on i386, but haven't figured out so far, see https://github.com/nim-lang/Nim/pull/17326

alaviss commented 3 years ago

I have 1 PR that fixes CI on i386 here: #17327 by upgrading to ubuntu-18.04' on i386

I would like to avoid this if possible.

alaviss commented 3 years ago

In the future I'll enginner a docker image as CI for nim, which should avoid all of these stupid issues with apt.

If you want to upgrade the ubuntu version @timotheecour, please upgrade it for all architectures.

timotheecour commented 3 years ago

=> replied in https://github.com/nim-lang/Nim/pull/17327#issuecomment-796396592

timotheecour commented 3 years ago

Re-opening because the hotfix https://github.com/nim-lang/Nim/pull/17327 worked for 2 days but the i386 linux problem re-occured even though its now using ubuntu 18.04

note that https://github.com/nim-lang/Nim/pull/17327 shouldn't be reverted since it does other useful things, but changing 18.04 back to 16.04 would be ok inside a fix for this issue
the set -e I added is correct (fail fast) and should be kept
see the different things i tried in https://github.com/nim-lang/Nim/pull/17326, none of which worked

/cc @alaviss since you've created the pipeline for i386 :-)

CI logs https://dev.azure.com/nim-lang/Nim/_build/results?buildId=14222&view=logs&jobId=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&j=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&t=38200048-f2b3-5d1c-03e3-3014d58a3656

sudo dpkg --add-architecture i386
Package: libllvm6.0
Pin: origin "azure.archive.ubuntu.com"
Pin-Priority: 1001

sudo apt-fast update -qq

sudo apt-fast install --no-install-recommends --allow-downgrades -yq g++-multilib gcc-multilib libcurl4-openssl-dev:i386 libgc-dev:i386 libsdl1.2-dev:i386 libsfml-dev:i386 libglib2.0-dev:i386 libffi-dev:i386
[apt-fast 17:19:44]
[apt-fast 17:19:44]Working... this may take a while.
E: Unable to correct problems, you have held broken packages.
[apt-fast 17:19:44]Package manager quit with exit code.
##[error]Bash exited with code '100'.
Finishing: Install dependencies (i386 Linux)

option 1

Self-hosted Linux agents, see https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops, or a docker image

option 2

docker image

links

Thorough CI Travis and Azure Pipelines scripts - Nim forum (doesn't seem useful unfortunately despite mentioning 32 bit linux)
no 32bit linux on the hosted agents: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml
https://github.com/inkstitch/inkstitch/issues/309#issuecomment-422808309

alaviss commented 3 years ago

@timotheecour try removing the manual pinning I made. That pinning was targeted for Ubuntu 16.04.

alaviss commented 3 years ago

@timotheecour try removing the manual pinning I made. That pinning was targeted for Ubuntu 16.04.

I've tried this but it doesn't seem to work.

Please disable 32 bit Linux testing for now, I'll build a container image later for CI.

timotheecour commented 3 years ago

I've tried this but it doesn't seem to work.

i'm pretty sure this is among the many things i've tried and IIRC it didnt' work; it was trying it again but azure is slow due to a temporary downtime refs https://status.dev.azure.com/ (EDIT: indeed I've checked again that this doesn't work)

disabling 32bit linux is pure cosmetics though, we could do it as last resort but I can do something better that just disables 3 tests, refs https://github.com/nim-lang/Nim/pull/17359

tests/async/tasyncssl.nim tests/stdlib/tssl.nim tests/niminaction/Chapter8/sfml/sfml_test.nim

see errors i'm getting here: https://github.com/nim-lang/Nim/pull/17359#issuecomment-797880252

the problem is right now the azure outage prevents further testing, hopefully resolved in a few minutes/hours

alaviss commented 3 years ago

Got a container image rolling: https://github.com/users/alaviss/packages/container/package/nim-ci

I haven't managed to get this working with Azure but you can see my ARM CI for the container in action: https://cloud.drone.io/alaviss/Nim/14/2/2

We will want to move our Linux CI image to this, given that it's a more controlled environment compared to the VM images provided by CI services.

timotheecour commented 3 years ago

re-opening this, sadly:

i386 fails again a

https://github.com/nim-lang/Nim/pull/17757

https://dev.azure.com/nim-lang/Nim/_build/results?buildId=15594&view=logs&jobId=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&j=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&t=38200048-f2b3-5d1c-03e3-3014d58a3656

sudo apt-fast update -qq

sudo apt-fast install --no-install-recommends --allow-downgrades -yq g++-multilib gcc-multilib libcurl4-openssl-dev:i386 libgc-dev:i386 libsdl1.2-dev:i386 libsfml-dev:i386 libglib2.0-dev:i386 libffi-dev:i386
[apt-fast 18:41:29]
[apt-fast 18:41:29]Working... this may take a while.
E: Unable to correct problems, you have held broken packages.
[apt-fast 18:41:29]Package manager quit with exit code.
##[error]Bash exited with code '100'.
Finishing: Install dependencies (i386 Linux)

probably another upstream issue; our setup for i386 emulation on azure is fragile indeed.

timotheecour commented 3 years ago

I disabled i386 again, see https://github.com/nim-lang/Nim/pull/17963

future work

this keeps happening so we need a more robust fix. 3 possible approaches worth exploring:

removing 1 dependency in i386 pipeline that causes issues; that's what i was doing in https://github.com/nim-lang/Nim/pull/17359 and it worked modulo just 3 failing tests (see also https://github.com/nim-lang/Nim/pull/17326)
docker based approach via https://github.com/timotheecour/Nim/pull/654
docker based approach via https://github.com/nim-lang/Nim/pull/17424, but see discussion in https://github.com/nim-lang/Nim/pull/17424#issuecomment-803382667

(all 3 are still WIP)

nim-lang / Nim