actions / runner-images

GitHub Actions runner images
MIT License
10.15k stars 3.06k forks source link

Ubuntu 22 build failing on Nginx script #7260

Closed nnellanspdl closed 1 year ago

nnellanspdl commented 1 year ago

Description

Trying to build the Ubuntu 22 image, using main branch. I've tried a few times, but keep getting the below error.

==> azure-arm.build_vhd: Provisioning with shell script: C:/a/1/s/runner-images/images/linux/scripts/installers/nginx.sh
    azure-arm.build_vhd: Reading package lists...
    azure-arm.build_vhd: Building dependency tree...
    azure-arm.build_vhd: Reading state information...
    azure-arm.build_vhd: You might want to run 'apt --fix-broken install' to correct these.
    azure-arm.build_vhd: The following packages have unmet dependencies:
    azure-arm.build_vhd:  nginx : Depends: nginx-core (< 1.18.0-6ubuntu14.3.1~) but it is not going to be installed or
    azure-arm.build_vhd:                   nginx-full (< 1.18.0-6ubuntu14.3.1~) but it is not going to be installed or
    azure-arm.build_vhd:                   nginx-light (< 1.18.0-6ubuntu14.3.1~) but it is not going to be installed or
    azure-arm.build_vhd:                   nginx-extras (< 1.18.0-6ubuntu14.3.1~) but it is not going to be installed
==> azure-arm.build_vhd: E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
    azure-arm.build_vhd:          Depends: nginx-core (>= 1.18.0-6ubuntu14.3) but it is not going to be installed or
    azure-arm.build_vhd:                   nginx-full (>= 1.18.0-6ubuntu14.3) but it is not going to be installed or
    azure-arm.build_vhd:                   nginx-light (>= 1.18.0-6ubuntu14.3) but it is not going to be installed or
    azure-arm.build_vhd:                   nginx-extras (>= 1.18.0-6ubuntu14.3) but it is not going to be installed
    azure-arm.build_vhd:  unixodbc : Depends: libodbc1 (>= 2.3.11-1) but it is not going to be installed
==> azure-arm.build_vhd: Failed to disable unit: Unit file nginx.service does not exist.
==> azure-arm.build_vhd: Script exited with non-zero exit status: 1. Allowed exit codes are: [0]
==> azure-arm.build_vhd: Step "StepProvision" failed

Platforms affected

Runner images affected

Image version and build link

n/a

Is it regression?

n/a

Expected behavior

Build to complete

Actual behavior

Build stalls at the nginx script

Repro steps

Use the scripts to try and create an Ubuntu 22 image.

ddobranic commented 1 year ago

Hello, @nnellanspdl! Thank you for reporting. After investigation I did not face any issues building new Ubuntu 22.04 image. Probably it was temporary issue. Could you please check again?

nnellanspdl commented 1 year ago

I tried another time this morning, and ran into the same error. So, not resolved for me.

nnellanspdl commented 1 year ago

This may have something to do with it?

There are errors with the mssql-cmd-tools.sh script and libodbc. The nginx script (the first error I posted above) is also complaining about libodbc. Could it be some sort of version conflict with those 2 items?

2023-03-13T14:31:40.4935378Z ==> azure-arm.build_vhd: Provisioning with shell script: C:/a/1/s/runner-images/images/linux/scripts/installers/mssql-cmd-tools.sh
2023-03-13T14:31:40.6977200Z     azure-arm.build_vhd: Hit:1 http://azure.archive.ubuntu.com/ubuntu jammy InRelease
2023-03-13T14:31:40.6990119Z     azure-arm.build_vhd: Get:2 http://azure.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
2023-03-13T14:31:40.7119041Z     azure-arm.build_vhd: Get:3 http://azure.archive.ubuntu.com/ubuntu jammy-backports InRelease [107 kB]
2023-03-13T14:31:40.7202659Z     azure-arm.build_vhd: Get:4 http://azure.archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
2023-03-13T14:31:40.7436582Z     azure-arm.build_vhd: Hit:5 https://packages.microsoft.com/ubuntu/18.04/prod bionic InRelease
2023-03-13T14:31:41.1104449Z     azure-arm.build_vhd: Hit:6 https://ppa.launchpadcontent.net/ubuntu-toolchain-r/test/ubuntu jammy InRelease
2023-03-13T14:31:42.1721390Z     azure-arm.build_vhd: Fetched 336 kB in 1s (615 kB/s)
2023-03-13T14:31:43.0714639Z     azure-arm.build_vhd: Reading package lists...
2023-03-13T14:31:43.1466211Z     azure-arm.build_vhd: Reading package lists...
2023-03-13T14:31:43.3605179Z     azure-arm.build_vhd: Building dependency tree...
2023-03-13T14:31:43.3616847Z     azure-arm.build_vhd: Reading state information...
2023-03-13T14:31:43.5399898Z     azure-arm.build_vhd: The following packages were automatically installed and are no longer required:
2023-03-13T14:31:43.5410804Z     azure-arm.build_vhd:   liblldb-12 liblldb-13
2023-03-13T14:31:43.5425308Z     azure-arm.build_vhd: Use 'sudo apt autoremove' to remove them.
2023-03-13T14:31:43.5441201Z     azure-arm.build_vhd: The following additional packages will be installed:
2023-03-13T14:31:43.5451395Z     azure-arm.build_vhd:   libodbc1 msodbcsql17 odbcinst odbcinst1debian2 unixodbc
2023-03-13T14:31:43.5461540Z     azure-arm.build_vhd: Suggested packages:
2023-03-13T14:31:43.5473307Z     azure-arm.build_vhd:   unixodbc-bin
2023-03-13T14:31:43.5648305Z     azure-arm.build_vhd: The following NEW packages will be installed:
2023-03-13T14:31:43.5666569Z     azure-arm.build_vhd:   libodbc1 msodbcsql17 mssql-tools odbcinst odbcinst1debian2 unixodbc
2023-03-13T14:31:43.5683260Z     azure-arm.build_vhd:   unixodbc-dev
2023-03-13T14:31:43.6386703Z     azure-arm.build_vhd: 0 upgraded, 7 newly installed, 0 to remove and 9 not upgraded.
2023-03-13T14:31:43.6399062Z     azure-arm.build_vhd: Need to get 1656 kB of archives.
2023-03-13T14:31:43.6411749Z     azure-arm.build_vhd: After this operation, 2798 kB of additional disk space will be used.
2023-03-13T14:31:43.6424915Z     azure-arm.build_vhd: Get:1 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 libodbc1 amd64 2.3.11-1 [486 kB]
2023-03-13T14:31:43.6840431Z     azure-arm.build_vhd: Get:2 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 odbcinst1debian2 amd64 2.3.11-1 [99.8 kB]
2023-03-13T14:31:43.6919386Z     azure-arm.build_vhd: Get:3 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 odbcinst amd64 2.3.11-1 [21.3 kB]
2023-03-13T14:31:43.6989795Z     azure-arm.build_vhd: Get:4 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 unixodbc amd64 2.3.11-1 [51.5 kB]
2023-03-13T14:31:43.7083688Z     azure-arm.build_vhd: Get:5 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 msodbcsql17 amd64 17.10.2.1-1 [745 kB]
2023-03-13T14:31:43.7288728Z     azure-arm.build_vhd: Get:6 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 mssql-tools amd64 17.10.1.1-1 [211 kB]
2023-03-13T14:31:43.7371918Z     azure-arm.build_vhd: Get:7 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 unixodbc-dev amd64 2.3.11-1 [42.4 kB]
2023-03-13T14:31:44.0405265Z     azure-arm.build_vhd: Preconfiguring packages ...
2023-03-13T14:31:44.3001556Z     azure-arm.build_vhd: Fetched 1656 kB in 0s (9953 kB/s)
2023-03-13T14:31:44.6238066Z     azure-arm.build_vhd: Selecting previously unselected package libodbc1:amd64.
2023-03-13T14:31:44.7189650Z     azure-arm.build_vhd: (Reading database ... 217882 files and directories currently installed.)
2023-03-13T14:31:44.7284920Z     azure-arm.build_vhd: Preparing to unpack .../0-libodbc1_2.3.11-1_amd64.deb ...
2023-03-13T14:31:44.8096847Z     azure-arm.build_vhd: Unpacking libodbc1:amd64 (2.3.11-1) ...
2023-03-13T14:31:44.8115131Z     azure-arm.build_vhd: dpkg: error processing archive /tmp/apt-dpkg-install-btyAoH/0-libodbc1_2.3.11-1_amd64.deb (--unpack):
2023-03-13T14:31:44.8129358Z     azure-arm.build_vhd:  trying to overwrite '/usr/lib/x86_64-linux-gnu/libodbc.so.2.0.0', which is also in package libodbc2:amd64 2.3.9-5
2023-03-13T14:31:44.8444194Z     azure-arm.build_vhd: dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
2023-03-13T14:31:45.1456984Z     azure-arm.build_vhd: Selecting previously unselected package odbcinst1debian2:amd64.
2023-03-13T14:31:45.1657335Z     azure-arm.build_vhd: Preparing to unpack .../1-odbcinst1debian2_2.3.11-1_amd64.deb ...
2023-03-13T14:31:45.1918509Z     azure-arm.build_vhd: Unpacking odbcinst1debian2:amd64 (2.3.11-1) ...
2023-03-13T14:31:45.4666123Z     azure-arm.build_vhd: Selecting previously unselected package odbcinst.
2023-03-13T14:31:45.4872090Z     azure-arm.build_vhd: Preparing to unpack .../2-odbcinst_2.3.11-1_amd64.deb ...
2023-03-13T14:31:45.5293791Z     azure-arm.build_vhd: Unpacking odbcinst (2.3.11-1) ...
2023-03-13T14:31:45.7295127Z     azure-arm.build_vhd: Selecting previously unselected package unixodbc.
2023-03-13T14:31:45.7521698Z     azure-arm.build_vhd: Preparing to unpack .../3-unixodbc_2.3.11-1_amd64.deb ...
2023-03-13T14:31:45.7792651Z     azure-arm.build_vhd: Unpacking unixodbc (2.3.11-1) ...
2023-03-13T14:31:46.0079463Z     azure-arm.build_vhd: Selecting previously unselected package msodbcsql17.
2023-03-13T14:31:46.0209180Z     azure-arm.build_vhd: Preparing to unpack .../4-msodbcsql17_17.10.2.1-1_amd64.deb ...
2023-03-13T14:31:47.2449159Z     azure-arm.build_vhd: Unpacking msodbcsql17 (17.10.2.1-1) ...
2023-03-13T14:31:48.2459715Z     azure-arm.build_vhd: Selecting previously unselected package mssql-tools.
2023-03-13T14:31:48.2461438Z     azure-arm.build_vhd: Preparing to unpack .../5-mssql-tools_17.10.1.1-1_amd64.deb ...
2023-03-13T14:31:48.2462432Z     azure-arm.build_vhd: Unpacking mssql-tools (17.10.1.1-1) ...
2023-03-13T14:31:48.2463781Z     azure-arm.build_vhd: Selecting previously unselected package unixodbc-dev.
2023-03-13T14:31:48.2464763Z     azure-arm.build_vhd: Preparing to unpack .../6-unixodbc-dev_2.3.11-1_amd64.deb ...
2023-03-13T14:31:48.2466403Z     azure-arm.build_vhd: Unpacking unixodbc-dev (2.3.11-1) ...
2023-03-13T14:31:48.2467467Z     azure-arm.build_vhd: Errors were encountered while processing:
2023-03-13T14:31:48.2468487Z     azure-arm.build_vhd:  /tmp/apt-dpkg-install-btyAoH/0-libodbc1_2.3.11-1_amd64.deb
2023-03-13T14:31:48.2469438Z     azure-arm.build_vhd: needrestart is being skipped since dpkg has failed
2023-03-13T14:31:48.4744504Z ==> azure-arm.build_vhd: E: Sub-process /usr/bin/dpkg returned an error code (1)
2023-03-13T14:31:48.5318708Z     azure-arm.build_vhd: Reading package lists...
2023-03-13T14:31:48.7383546Z     azure-arm.build_vhd: Building dependency tree...
2023-03-13T14:31:48.7391859Z     azure-arm.build_vhd: Reading state information...
2023-03-13T14:31:48.8011995Z     azure-arm.build_vhd: Correcting dependencies... Done
2023-03-13T14:31:48.9549573Z     azure-arm.build_vhd: The following packages were automatically installed and are no longer required:
2023-03-13T14:31:48.9557490Z     azure-arm.build_vhd:   liblldb-12 liblldb-13
2023-03-13T14:31:48.9570570Z     azure-arm.build_vhd: Use 'sudo apt autoremove' to remove them.
2023-03-13T14:31:48.9595451Z     azure-arm.build_vhd: The following additional packages will be installed:
2023-03-13T14:31:48.9612225Z     azure-arm.build_vhd:   libodbc1
2023-03-13T14:31:48.9641938Z     azure-arm.build_vhd: Suggested packages:
2023-03-13T14:31:48.9656011Z     azure-arm.build_vhd:   unixodbc-bin
2023-03-13T14:31:48.9689621Z     azure-arm.build_vhd: The following NEW packages will be installed:
2023-03-13T14:31:48.9695906Z     azure-arm.build_vhd:   libodbc1
2023-03-13T14:31:49.3030814Z     azure-arm.build_vhd: 0 upgraded, 1 newly installed, 0 to remove and 9 not upgraded.
2023-03-13T14:31:49.3036693Z     azure-arm.build_vhd: 6 not fully installed or removed.
2023-03-13T14:31:49.3048866Z     azure-arm.build_vhd: Need to get 0 B/486 kB of archives.
2023-03-13T14:31:49.3058875Z     azure-arm.build_vhd: After this operation, 623 kB of additional disk space will be used.
2023-03-13T14:31:51.2853627Z     azure-arm.build_vhd: (Reading database ... 217959 files and directories currently installed.)
2023-03-13T14:31:51.2963065Z     azure-arm.build_vhd: Preparing to unpack .../libodbc1_2.3.11-1_amd64.deb ...
2023-03-13T14:31:51.3223879Z     azure-arm.build_vhd: Unpacking libodbc1:amd64 (2.3.11-1) ...
2023-03-13T14:31:51.3251522Z     azure-arm.build_vhd: dpkg: error processing archive /var/cache/apt/archives/libodbc1_2.3.11-1_amd64.deb (--unpack):
2023-03-13T14:31:51.3265985Z     azure-arm.build_vhd:  trying to overwrite '/usr/lib/x86_64-linux-gnu/libodbc.so.2.0.0', which is also in package libodbc2:amd64 2.3.9-5
2023-03-13T14:31:51.3276585Z     azure-arm.build_vhd: dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
2023-03-13T14:31:51.5239972Z     azure-arm.build_vhd: Errors were encountered while processing:
2023-03-13T14:31:51.5251654Z     azure-arm.build_vhd:  /var/cache/apt/archives/libodbc1_2.3.11-1_amd64.deb
2023-03-13T14:31:51.6812255Z     azure-arm.build_vhd: needrestart is being skipped since dpkg has failed
2023-03-13T14:31:52.0790610Z ==> azure-arm.build_vhd: E: Sub-process /usr/bin/dpkg returned an error code (1)
2023-03-13T14:31:57.0795157Z     azure-arm.build_vhd: Pester v5.4.0
2023-03-13T14:31:57.0964234Z     azure-arm.build_vhd:
2023-03-13T14:31:57.0975645Z     azure-arm.build_vhd: Starting discovery in 1 files.
2023-03-13T14:31:57.6506234Z     azure-arm.build_vhd: Discovery found 100 tests in 512ms.
2023-03-13T14:31:57.6605935Z     azure-arm.build_vhd: Filter 'FullName' set to ('MSSQLCommandLineTools').
2023-03-13T14:31:57.6617377Z     azure-arm.build_vhd: Filters selected 1 tests to run.
2023-03-13T14:31:57.6695049Z     azure-arm.build_vhd: Running tests.
2023-03-13T14:31:57.6809412Z     azure-arm.build_vhd:
2023-03-13T14:31:57.6825724Z     azure-arm.build_vhd: Running tests from '/imagegeneration/tests/Tools.Tests.ps1'
2023-03-13T14:31:57.7883781Z     azure-arm.build_vhd: Describing MSSQLCommandLineTools
2023-03-13T14:31:57.9805362Z     azure-arm.build_vhd:   [+] sqlcmd 170ms (120ms|50ms)
2023-03-13T14:31:58.2027652Z     azure-arm.build_vhd: Tests completed in 909ms
2023-03-13T14:31:58.2095516Z     azure-arm.build_vhd: Tests Passed: 1, Failed: 0, Skipped: 0 NotRun: 99
ddobranic commented 1 year ago

We don't have such an issue when building new Ubuntu 22.04 image. Here is the part of the log:

    azure-arm.build_vhd: The following additional packages will be installed:
    azure-arm.build_vhd:   libmysqlclient21 libodbc2 libpq5 libre2-9 mysql-common
    azure-arm.build_vhd: Suggested packages:
    azure-arm.build_vhd:   odbc-postgresql tdsodbc
    azure-arm.build_vhd: The following NEW packages will be installed:
    azure-arm.build_vhd:   libmysqlclient21 libodbc2 libpq5 libre2-9 mysql-common sphinxsearch
    azure-arm.build_vhd: 0 upgraded, 6 newly installed, 0 to remove and 1 not upgraded.

and

   ==> azure-arm.build_vhd: Provisioning with shell script: C:/a/_temp/image-gen/images/linux/scripts/installers/mssql-cmd-tools.sh
    azure-arm.build_vhd: Hit:1 http://azure.archive.ubuntu.com/ubuntu jammy InRelease
    azure-arm.build_vhd: Get:2 http://azure.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
    azure-arm.build_vhd: Get:3 http://azure.archive.ubuntu.com/ubuntu jammy-backports InRelease [107 kB]
    azure-arm.build_vhd: Get:4 http://azure.archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
    azure-arm.build_vhd: Hit:5 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease
    azure-arm.build_vhd: Hit:6 https://ppa.launchpadcontent.net/ubuntu-toolchain-r/test/ubuntu jammy InRelease
    azure-arm.build_vhd: Fetched 336 kB in 1s (448 kB/s)
    azure-arm.build_vhd: Reading package lists...
    azure-arm.build_vhd: Reading package lists...
    azure-arm.build_vhd: Building dependency tree...
    azure-arm.build_vhd: Reading state information...
    azure-arm.build_vhd: The following packages were automatically installed and are no longer required:
    azure-arm.build_vhd:   liblldb-12 liblldb-13
    azure-arm.build_vhd: Use 'sudo apt autoremove' to remove them.
    azure-arm.build_vhd: The following additional packages will be installed:
    azure-arm.build_vhd:   libodbccr2 libodbcinst2 msodbcsql17 odbcinst unixodbc unixodbc-common
    azure-arm.build_vhd: The following NEW packages will be installed:
    azure-arm.build_vhd:   libodbccr2 libodbcinst2 msodbcsql17 mssql-tools odbcinst unixodbc
    azure-arm.build_vhd:   unixodbc-common unixodbc-dev
    azure-arm.build_vhd: 0 upgraded, 8 newly installed, 0 to remove and 9 not upgraded.
nnellanspdl commented 1 year ago

Your mssql-cmd-tools.sh wants to do this:

azure-arm.build_vhd: The following additional packages will be installed:
azure-arm.build_vhd:   libmysqlclient21 libodbc2 libpq5 libre2-9 mysql-common
azure-arm.build_vhd: Suggested packages:
azure-arm.build_vhd:   odbc-postgresql tdsodbc
azure-arm.build_vhd: The following NEW packages will be installed:
azure-arm.build_vhd:   libmysqlclient21 libodbc2 libpq5 libre2-9 mysql-common sphinxsearch

Mine wants to do this:

azure-arm.build_vhd: The following additional packages will be installed:
azure-arm.build_vhd:   libodbc1 msodbcsql17 odbcinst odbcinst1debian2 unixodbc
azure-arm.build_vhd: Suggested packages:
azure-arm.build_vhd:   unixodbc-bin
azure-arm.build_vhd: The following NEW packages will be installed:
azure-arm.build_vhd:   libodbc1 msodbcsql17 mssql-tools odbcinst odbcinst1debian2 unixodbc
azure-arm.build_vhd:   unixodbc-dev
ddobranic commented 1 year ago

In you previous log, I can see:

   2023-03-13T14:31:40.7436582Z     azure-arm.build_vhd: Hit:5 https://packages.microsoft.com/ubuntu/18.04/prod bionic InRelease

Using Ubuntu 18.04 repo may be the reason for using old version and dependency clash.

nnellanspdl commented 1 year ago

I'm using the official build scripts. I'm not making changes to repos.

ddobranic commented 1 year ago

Somehow it is there. Could you please checkout clean repo and try from scratch?

nnellanspdl commented 1 year ago

Again, I am not making manual changes. I am downloading your repo and running your powershell script. That's it.

My pipelines run on ephemeral runners, they are destroyed after they run a job. So, my environment is always from scratch in that respect.

jshield commented 1 year ago

this appears to be as a result of some sort of daemonic influence replacing the ms repository apt reference with bionic's, i'm currently investigating potential apotropaic solutions that will dissuade the evil taking take root during the build.

i'm referring to MdE or something along those lines

mikhailkoliada commented 1 year ago

Hey! It is purely connected to https://github.com/microsoft/linux-package-repositories/issues/39 as packages intersect between repos with different apt priority

jshield commented 1 year ago

I was able to confirm that disabling MdE provisioning prevents the jammy reference being replaced by bionic for me.

I noticed it would happen at random times during the build, one step would update 22.04, then the next would try 18.04 and apt had locks taken when it tried, suggesting that something outside the packer build was trying to install software.

setting ExcludeMdEAutoProvisioning tag to True has completely stopped this from happening, thus I can only conclude that MdE provisioning script doesn't detect jammy correctly or is forcing it to use bionic repos.