microsoft / linux-package-repositories

Microsoft Packaged Linux Software (DEBs, RPMs, etc) are hosted on packages.microsoft.com (PMC) made available as native Linux repositories for use with package managers like APT, YUM, etc.
https://packages.microsoft.com
MIT License
64 stars 17 forks source link

Ubuntu 22.04: unattended-upgrades fought with ubuntu repo and broke my dotnet install #152

Open Skyedra opened 1 month ago

Skyedra commented 1 month ago

Today, after an automatic reboot for a kernel update, I woke up to find a dotnet-related unattended upgrade had clobbered one of my dotnet-powered servers:

Architecture: x64
Framework: 'Microsoft.NETCore.App', version '8.0.0' (x64)
.NET location: /usr/share/dotnet

No frameworks were found.

It seems like what happened is during an automatic update, apt pulled the updated ubuntu-provided package instead of the microsoft-provided package and that left dotnet in a broken state until a manual update fixed things.

Regardless, I quickly realized the problem was the suddenly missing deps package and a simple apt dist-upgrade fixed it, but I thought it worth raising in case I have things misconfigured or otherwise would like to avoid having a similar issue in future.

Logs

In /var/log/unattended-upgrades/unattended-upgrades.log I can see the problem seems to be dotnet-runtime-deps-8.0 got auto-removed for some reason. Possibly it is due to dotnet-hostfxr-8.0 being kept back for some reason:

2024-07-11 06:06:10,993 INFO Starting unattended upgrades script
2024-07-11 06:06:10,993 INFO Allowed origins are: o=Ubuntu,a=jammy, o=Ubuntu,a=jammy-security, o=UbuntuESMApps,a=jammy-apps-security, o=UbuntuESM,a=jammy-inf
ra-security
2024-07-11 06:06:10,994 INFO Initial blacklist: 
2024-07-11 06:06:10,994 INFO Initial whitelist (not strict): 
2024-07-11 06:06:12,906 INFO Packages that will be upgraded: aspnetcore-runtime-8.0 aspnetcore-targeting-pack-8.0 dotnet-apphost-pack-8.0 dotnet-runtime-8.0 
dotnet-targeting-pack-8.0 libpython3.10 libpython3.10-dev libpython3.10-minimal libpython3.10-stdlib netstandard-targeting-pack-2.1 python3.10 python3.10-dev
 python3.10-minimal
2024-07-11 06:06:12,906 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2024-07-11 06:06:34,038 INFO All upgrades installed
2024-07-11 06:06:36,946 INFO Packages that were successfully auto-removed: dotnet-runtime-deps-8.0
2024-07-11 06:06:36,946 INFO Packages that are kept back: 
2024-07-11 06:06:37,013 INFO Package dotnet-hostfxr-8.0 is kept back because a related package is kept back or due to local apt_preferences(5).
2024-07-12 06:10:02,418 INFO Starting unattended upgrades script
2024-07-12 06:10:02,419 INFO Allowed origins are: o=Ubuntu,a=jammy, o=Ubuntu,a=jammy-security, o=UbuntuESMApps,a=jammy-apps-security, o=UbuntuESM,a=jammy-infra-security
2024-07-12 06:10:02,419 INFO Initial blacklist: 
2024-07-12 06:10:02,419 INFO Initial whitelist (not strict): 
2024-07-12 06:10:03,279 INFO No packages found that can be upgraded unattended and no pending auto-removals
2024-07-12 06:10:03,287 INFO Package dotnet-hostfxr-8.0 is kept back because a related package is kept back or due to local apt_preferences(5).
2024-07-12 11:51:50,079 INFO Starting unattended upgrades script
2024-07-12 11:51:50,080 INFO Allowed origins are: o=Ubuntu,a=jammy, o=Ubuntu,a=jammy-security, o=UbuntuESMApps,a=jammy-apps-security, o=UbuntuESM,a=jammy-infra-security
2024-07-12 11:51:50,080 INFO Initial blacklist: 
2024-07-12 11:51:50,080 INFO Initial whitelist (not strict): 
2024-07-13 06:01:07,885 INFO Starting unattended upgrades script
2024-07-13 06:01:07,885 INFO Allowed origins are: o=Ubuntu,a=jammy, o=Ubuntu,a=jammy-security, o=UbuntuESMApps,a=jammy-apps-security, o=UbuntuESM,a=jammy-infra-security
2024-07-13 06:01:07,885 INFO Initial blacklist: 
2024-07-13 06:01:07,885 INFO Initial whitelist (not strict): 
2024-07-13 06:01:08,987 INFO Packages that will be upgraded: apache2-utils
2024-07-13 06:01:08,987 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2024-07-13 06:01:12,376 INFO All upgrades installed
2024-07-13 06:01:12,549 INFO Package dotnet-hostfxr-8.0 is kept back because a related package is kept back or due to local apt_preferences(5).
2024-07-13 11:43:03,805 INFO Starting unattended upgrades script
2024-07-13 11:43:03,805 INFO Allowed origins are: o=Ubuntu,a=jammy, o=Ubuntu,a=jammy-security, o=UbuntuESMApps,a=jammy-apps-security, o=UbuntuESM,a=jammy-infra-security

/var/log/apt/history.log provides some additional information:

Start-Date: 2024-07-11  06:06:18
Commandline: /usr/bin/unattended-upgrade
Upgrade: aspnetcore-runtime-8.0:amd64 (8.0.6-1, 8.0.7-0ubuntu1~22.04.1)
End-Date: 2024-07-11  06:06:18

Start-Date: 2024-07-11  06:06:20
Commandline: /usr/bin/unattended-upgrade
Upgrade: dotnet-apphost-pack-8.0:amd64 (8.0.6-1, 8.0.7-0ubuntu1~22.04.1)
End-Date: 2024-07-11  06:06:21

Start-Date: 2024-07-11  06:06:23
Commandline: /usr/bin/unattended-upgrade
Upgrade: netstandard-targeting-pack-2.1:amd64 (6.0.128-0ubuntu1~22.04.2, 6.0.132-0ubuntu1~22.04.1)
End-Date: 2024-07-11  06:06:23

Start-Date: 2024-07-11  06:06:26
Commandline: /usr/bin/unattended-upgrade
Upgrade: aspnetcore-targeting-pack-8.0:amd64 (8.0.6-1, 8.0.7-0ubuntu1~22.04.1)
End-Date: 2024-07-11  06:06:26

Start-Date: 2024-07-11  06:06:29
Commandline: /usr/bin/unattended-upgrade
Upgrade: dotnet-runtime-8.0:amd64 (8.0.6-1, 8.0.7-0ubuntu1~22.04.1)
End-Date: 2024-07-11  06:06:29

Start-Date: 2024-07-11  06:06:32
Commandline: /usr/bin/unattended-upgrade
Upgrade: dotnet-targeting-pack-8.0:amd64 (8.0.6-1, 8.0.7-0ubuntu1~22.04.1)
End-Date: 2024-07-11  06:06:32

Start-Date: 2024-07-11  06:06:35
Commandline: /usr/bin/unattended-upgrade
Remove: dotnet-runtime-deps-8.0:amd64 (8.0.6-1)
End-Date: 2024-07-11  06:06:35

Start-Date: 2024-07-13  06:01:10
Commandline: /usr/bin/unattended-upgrade
Upgrade: apache2-utils:amd64 (2.4.52-1ubuntu4.10, 2.4.52-1ubuntu4.11)
End-Date: 2024-07-13  06:01:10

Start-Date: 2024-07-17  06:46:01
Commandline: /usr/bin/unattended-upgrade
Upgrade: linux-libc-dev:amd64 (5.15.0-113.123, 5.15.0-116.126)
End-Date: 2024-07-17  06:46:02

Start-Date: 2024-07-17  06:46:05
Commandline: /usr/bin/unattended-upgrade
Install: linux-headers-5.15.0-116:amd64 (5.15.0-116.126, automatic), linux-modules-extra-5.15.0-116-generic:amd64 (5.15.0-116.126, automatic), linux-headers-5.15.0-116-generic:amd64 (5.15.0-116.126, automatic), linux-image-5.15.0-116-generic:amd64 (5.15.0-116.126, automatic), linux-modules-5.15.0-116-generic:amd64 (5.15.0-116.126, automatic)
Upgrade: linux-headers-generic:amd64 (5.15.0.113.113, 5.15.0.116.116), linux-generic:amd64 (5.15.0.113.113, 5.15.0.116.116), linux-image-generic:amd64 (5.15.0.113.113, 5.15.0.116.116)
End-Date: 2024-07-17  06:46:26

Start-Date: 2024-07-17  14:20:40
Commandline: apt-get dist-upgrade
Install: dotnet-runtime-deps-8.0:amd64 (8.0.7-1, automatic)
Upgrade: aspnetcore-runtime-8.0:amd64 (8.0.7-0ubuntu1~22.04.1, 8.0.7-1), tzdata:amd64 (2024a-0ubuntu0.22.04, 2024a-0ubuntu0.22.04.1), libldap-common:amd64 (2.5.17+dfsg-0ubuntu0.22.04.1, 2.5.18+dfsg-0ubuntu0.22.04.1), dotnet-hostfxr-8.0:amd64 (8.0.6-1, 8.0.7-1), dotnet-sdk-8.0:amd64 (8.0.301-1, 8.0.303-1), dotnet-targeting-pack-8.0:amd64 (8.0.7-0ubuntu1~22.04.1, 8.0.7-1), landscape-common:amd64 (23.02-0ubuntu1~22.04.2, 23.02-0ubuntu1~22.04.3), libldap-2.5-0:amd64 (2.5.17+dfsg-0ubuntu0.22.04.1, 2.5.18+dfsg-0ubuntu0.22.04.1), ubuntu-pro-client-l10n:amd64 (32.3~22.04, 32.3.1~22.04), cloud-init:amd64 (24.1.3-0ubuntu1~22.04.4, 24.1.3-0ubuntu1~22.04.5), ubuntu-server-minimal:amd64 (1.481.1, 1.481.2), snapd:amd64 (2.62+22.04, 2.63+22.04), dotnet-host:amd64 (8.0.6-1, 8.0.7-1), linux-firmware:amd64 (20220329.git681281e4-0ubuntu3.30, 20220329.git681281e4-0ubuntu3.31), ubuntu-standard:amd64 (1.481.1, 1.481.2), ubuntu-server:amd64 (1.481.1, 1.481.2), dotnet-apphost-pack-8.0:amd64 (8.0.7-0ubuntu1~22.04.1, 8.0.7-1), aspnetcore-targeting-pack-8.0:amd64 (8.0.7-0ubuntu1~22.04.1, 8.0.7-1), ubuntu-advantage-tools:amd64 (32.3~22.04, 32.3.1~22.04), ubuntu-minimal:amd64 (1.481.1, 1.481.2), dotnet-runtime-8.0:amd64 (8.0.7-0ubuntu1~22.04.1, 8.0.7-1), ubuntu-pro-client:amd64 (32.3~22.04, 32.3.1~22.04)
End-Date: 2024-07-17  14:21:36

And to confirm the source of the package:

# apt policy dotnet-runtime-8.0
dotnet-runtime-8.0:
  Installed: 8.0.7-1
  Candidate: 8.0.7-1
  Version table:
 *** 8.0.7-1 500
        500 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     8.0.7-0ubuntu1~22.04.1 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     8.0.6-1 500
        500 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 Packages
     8.0.5-1 500
        500 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 Packages
     8.0.4-1 500
        500 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 Packages
     8.0.3-1 500
        500 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 Packages
     8.0.2-1 500
        500 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 Packages
     8.0.1-1 500
        500 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 Packages
     8.0.0-1 500
        500 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 Packages

Possible Solution

Two things I've gone ahead and tried:

1) I've gone ahead and followed the instructions here to deprioritize the ubuntu feed for dotnet.

2) I also realize I had never added the microsoft repo to my /etc/apt/apt.conf.d/50unattended-upgrades, so I've also done that per this answer.

If that sounds wrong or there's something else I should be doing instead, please let me know!