Ubuntu repositories broken

stefanjarina commented 3 years ago

Hello,

Ubuntu repositories seems to be broken, specifically I am getting '404 not found' for any .deb

from apt update

Err:1 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 dotnet-host amd64 5.0.7-1
  404  Not Found [IP: 104.214.230.139 443]
Err:2 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 dotnet-hostfxr-3.1 amd64 3.1.16-1
  404  Not Found [IP: 104.214.230.139 443]
Err:3 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 dotnet-runtime-deps-3.1 amd64 3.1.16-1
  404  Not Found [IP: 104.214.230.139 443]
Err:4 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 dotnet-runtime-3.1 amd64 3.1.16-1
  404  Not Found [IP: 104.214.230.139 443]
Err:5 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 aspnetcore-runtime-3.1 amd64 3.1.16-1
  404  Not Found [IP: 104.214.230.139 443]
Err:6 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 dotnet-apphost-pack-3.1 amd64 3.1.16-1
  404  Not Found [IP: 104.214.230.139 443]
Err:7 https://packages.microsoft.com/ubuntu/20.04/prod focal/main amd64 dotnet-sdk-3.1 amd64 3.1.410-1
  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/d/dotnet-host/dotnet-host-5.0.7-x64.deb  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/d/dotnet-hostfxr-3.1/dotnet-hostfxr-3.1.16-x64.deb  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/d/dotnet-runtime-deps-3.1/dotnet-runtime-deps-3.1.16-x64.deb  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/d/dotnet-runtime-3.1/dotnet-runtime-3.1.16-x64.deb  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/a/aspnetcore-runtime-3.1/aspnetcore-runtime-3.1.16-x64.deb  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/d/dotnet-apphost-pack-3.1/dotnet-apphost-pack-3.1.16-x64.deb  404  Not Found [IP: 104.214.230.139 443]
E: Failed to fetch https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/d/dotnet-sdk-3.1/dotnet-sdk-3.1.410-x64.deb  404  Not Found [IP: 104.214.230.139 443]

Tried to re-download the repository package, however it seems to be simply not there:

~> wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
--2021-06-16 22:40:24--  https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb
Resolving packages.microsoft.com (packages.microsoft.com)... 13.80.99.124
Connecting to packages.microsoft.com (packages.microsoft.com)|13.80.99.124|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-06-16 22:40:24 ERROR 404: Not Found.

This issue seems to be there only for .deb files though, as others can be downloaded without issues, e.g.

~> wget https://packages.microsoft.com/config/ubuntu/20.04/mssql-server-2019.list -O mssql-server-2019.list
--2021-06-16 22:52:28--  https://packages.microsoft.com/config/ubuntu/20.04/mssql-server-2019.list
Resolving packages.microsoft.com (packages.microsoft.com)... 40.114.136.21
Connecting to packages.microsoft.com (packages.microsoft.com)|40.114.136.21|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 101 [application/octet-stream]
Saving to: ‘mssql-server-2019.list’

mssql-server-2019.list        100%[=================================================>]     101  --.-KB/s    in 0s

2021-06-16 22:52:28 (29.2 MB/s) - ‘mssql-server-2019.list’ saved [101/101]

~> cat mssql-server-2019.list
deb [arch=amd64,armhf,arm64] https://packages.microsoft.com/ubuntu/20.04/mssql-server-2019 focal main

This is true for ALL Ubuntu versions

bquantump commented 3 years ago

Broken here too

wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb --2021-06-16 20:58:37-- https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb Resolving packages.microsoft.com (packages.microsoft.com)... 13.66.21.183 Connecting to packages.microsoft.com (packages.microsoft.com)|13.66.21.183|:443... connected. HTTP request sent, awaiting response... 404 Not Found

charliemenke commented 3 years ago

Seeing similar issue for: Failed to fetch https://packages.microsoft.com/debian/10/prod/pool/main/m/msodbcsql17/msodbcsql17_17.7.2.1-1_amd64.deb 404 Not Found [IP: 13.84.47.71 443]

akiehl2000 commented 3 years ago

Same here:

michaelgsharp commented 3 years ago

@rbhanda are you able to help here? This seems similar to the situation we saw the other day #6371

rbhanda commented 3 years ago

Thanks for letting me @michaelgsharp . I am following up on this with our infrastructure team.

rbhanda commented 3 years ago

Update: Our infra team is working on this. There is an issue with some of the mirrors on packages.microsoft.com so as per them, the current ETA to resolve this issue is in next two hours or so.

SudhirNune commented 3 years ago

@rbhanda - can you please provide me an update ?

rbhanda commented 3 years ago

Update: our infra team is still working to resolve this issue. They ran into some space issues but this issue should be resolved quickly. Unfortunately, I do not have an ETA yet.

kitsuyui commented 3 years ago

I also understand this is little-bit off-topic for this project.

But let me just say this; I think the problem is not only package repository was broken. Another side of the problem is about communication.

It might be there some progresses on some task tracking system ore some collaboration tools at Microsoft. But we can't see it.

Following issues were opened and some issues were closed so quickly. It doesn't show somewhere to go. It doesn't show how long we must wait. So issues are opened repeatedly here and there. This is spreading between projects outside of Microsoft.

Niek commented 3 years ago

FYI the Edge repo is broken as well, the whole pool directory is gone: https://packages.microsoft.com/repos/edge/pool/

phknot commented 3 years ago

FYI the Edge repo is broken as well, the whole pool directory is gone: https://packages.microsoft.com/repos/edge/pool/

Same problem for the Teams repo

gdams commented 3 years ago

We've also had the bug raised against our Microsoft build of OpenJDK repo https://github.com/microsoft/openjdk/discussions/74

r3volver commented 3 years ago

I can confirm this issue. 404 in every debian or ubuntu .deb I search.

In the meantime, is there a mirror where we could find the packages?

Get:1 http://deb.debian.org/debian buster/main amd64 multiarch-support amd64 2.28-10 [215 kB] Ign:2 https://packages.microsoft.com/debian/10/prod buster/main amd64 odbcinst amd64 2.3.7 Ign:3 https://packages.microsoft.com/debian/10/prod buster/main amd64 unixodbc-dev amd64 2.3.7 Ign:4 https://packages.microsoft.com/debian/10/prod buster/main amd64 odbcinst1debian2 amd64 2.3.7 Ign:5 https://packages.microsoft.com/debian/10/prod buster/main amd64 libodbc1 amd64 2.3.7 Ign:6 https://packages.microsoft.com/debian/10/prod buster/main amd64 unixodbc amd64 2.3.7 Err:7 https://packages.microsoft.com/debian/10/prod buster/main amd64 msodbcsql17 amd64 17.7.2.1-1 404 Not Found [IP: 40.114.136.21 443] Err:2 https://packages.microsoft.com/debian/10/prod buster/main amd64 odbcinst amd64 2.3.7 404 Not Found [IP: 40.114.136.21 443] Err:3 https://packages.microsoft.com/debian/10/prod buster/main amd64 unixodbc-dev amd64 2.3.7 404 Not Found [IP: 40.114.136.21 443] Err:4 https://packages.microsoft.com/debian/10/prod buster/main amd64 odbcinst1debian2 amd64 2.3.7 404 Not Found [IP: 40.114.136.21 443] Err:5 https://packages.microsoft.com/debian/10/prod buster/main amd64 libodbc1 amd64 2.3.7 404 Not Found [IP: 40.114.136.21 443] Err:6 https://packages.microsoft.com/debian/10/prod buster/main amd64 unixodbc amd64 2.3.7 404 Not Found [IP: 40.114.136.21 443] Fetched 215 kB in 1s (373 kB/s) Unable to correct missing packages. E: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/pool/main/u/unixodbc/odbcinst_2.3.7_amd64.deb 404 Not Found [IP: 40.114.136.21 443] E: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/pool/main/u/unixodbc/unixodbc-dev_2.3.7_amd64.deb 404 Not Found [IP: 40.114.136.21 443] E: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/pool/main/u/unixodbc/odbcinst1debian2_2.3.7_amd64.deb 404 Not Found [IP: 40.114.136.21 443] E: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/pool/main/u/unixodbc/libodbc1_2.3.7_amd64.deb 404 Not Found [IP: 40.114.136.21 443] E: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/pool/main/u/unixodbc/unixodbc_2.3.7_amd64.deb 404 Not Found [IP: 40.114.136.21 443] E: Failed to fetch https://packages.microsoft.com/debian/10/prod/pool/main/m/msodbcsql17/msodbcsql17_17.7.2.1-1_amd64.deb 404 Not Found [IP: 40.114.136.21 443] E: Aborting install.

hussaynv commented 3 years ago

Some people suggesting its an issue isolated to Europe, other regions as able to build https://docs.microsoft.com/en-us/answers/questions/439491/pyodbc-install-error-in-azure-databricks-file-not.html

sciyako commented 3 years ago

14 hours! this is blocking us in the production environment right now. as we can't build new machines/vms in Azure.

GiaNTizmO commented 3 years ago

same too with Debain10 and Ubuntu 20.04LTS

gmemstr commented 3 years ago

Might be worth holding off on the "me too!" or "+1" comments unless there's some new information to make the issue easier to follow.

Albonycal commented 3 years ago

Space issues? LOL

tomas-forsman commented 3 years ago

Ask Bill to buy some disks from Amazon ;-)

SudhirNune commented 3 years ago

I started seeing issue with Certificates (https://packages.microsoft.com/keys/microsoft.asc) too

Anmol007 commented 3 years ago

I can confirm this issue too. We noticed this today morning and the releases are blocked because of this.

command in DockerFile: RUN apt-get update && apt-get install -y --no-install-recommends \ g++ \ unixodbc-dev

Error in azure pipelines: Err:1 https://packages.microsoft.com/debian/9/prod stretch/main amd64 unixodbc-dev amd64 2.3.7 404 Not Found [IP: 137.135.134.16 443] E: Failed to fetch https://packages.microsoft.com/debian/9/prod/pool/main/u/unixodbc/unixodbc-dev_2.3.7_amd64.deb 404 Not Found [IP: 137.135.134.16 443]

aashu-kumari-by commented 3 years ago

I have also started seeing this issue from 17th june,2021 morning IST timezone, Need to get 210 MB of archives. After this operation, 0 B of additional disk space will be used. Get:1 https://packages.microsoft.com/repos/microsoft-ubuntu-focal-prod focal/main amd64 azure-functions-core-tools-3 amd64 3.0.3568-1 [210 MB] It has got stuck here and not able to get the response. Kindly help in resolving the issue.

This one also i got - curl: (28) Failed to connect to packages.microsoft.com port 443: Connection timed out gpg: no valid OpenPGP data found. Error: Process completed with exit code 2.

m17kea commented 3 years ago

@rbhanda any update on this, this is blocking production server builds for us

SudhirNune commented 3 years ago

@rbhanda - its same here and I hope a lot of processes / projects are impacted, is there any update ETA, and any MSFT official status page having the details and update ?

AymanAlneser commented 3 years ago

any updates on this this is affecting Production environment and we need some ETA to communicate to our Stack holders

rvResolving commented 3 years ago

for people who need, snap installation is working, i know it is not apt but it can be a temporary solution

Kyle2123 commented 3 years ago

14 hours later and this is still broken - prevents rolling new builds. Can we get an update on this please???

tappoz commented 3 years ago

for people who need, snap installation is working, i know it is not apt but it can be a temporary solution

@rvResolving can you please provide some context with commands around what works for you?

I am using the apt repositories (on Debian) for the SQL Server integration, so I tried this:

snap find microsoft | grep -e msodbcsql17 -e mssql-tools -e unixodbc-dev

It does not find anything. I also tried the strings msodbcsql17 and so on, still nothing is being found.

coding-liki commented 3 years ago

All deb-based repos at packages.microsoft.com are broken. I suggest, that they hid the "pool" folder for some reason.

rpyndus commented 3 years ago

I tried to use proxy in US, but unfortunately no success in doing that, pool folder does not exists. Also snap wont help for packages aziot-edge and moby-engine

oising commented 3 years ago

Ask Bill to buy some disks from Amazon ;-)

No need! His 5G vaccine is turning everybody with a phone into supplementary Azure bio-blob storage! I wonder what files I'm infected with? Exciting!

rvResolving commented 3 years ago

It does not find anything. I also tried the strings msodbcsql17 and so on, still nothing is being found.

@tappoz i don't know for msodbcsql17 i need the donet-sdk

and I just do sudo snap install dotnet-sdk --classic --channel=3.1

I launch it with ansible but this is the final command I check for msodbcsql17 but I think you can't do what I do for dotnet

tappoz commented 3 years ago

There seems to be a common root cause to all the Microsoft apt repos. People is cross-referencing different GitHub issues in different git repositories.

schoolytics-dev0 commented 3 years ago

Similar issues here trying to use Ubuntu packages

felipeschneider88 commented 3 years ago

I was able to get the package using Invoke-WebRequest

 iwr https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -OutFile packages-microsoft-prod.deb

dir | findstr "pac"
-a----         6/17/2021  10:31 AM           3124 packages-microsoft-prod.deb

DLewisEndeavor commented 3 years ago

This seems to be the most active thread on the issue, does anyone know if this is being tracked elsewhere by Microsoft, this is affecting our production systems and we need some sort of ETA.

aashu-kumari-by commented 3 years ago

I have also started seeing this issue from 17th june,2021 morning IST timezone, Need to get 210 MB of archives. After this operation, 0 B of additional disk space will be used. Get:1 https://packages.microsoft.com/repos/microsoft-ubuntu-focal-prod focal/main amd64 azure-functions-core-tools-3 amd64 3.0.3568-1 [210 MB] It has got stuck here and not able to get the response. Kindly help in resolving the issue.

This one also i got - curl: (28) Failed to connect to packages.microsoft.com port 443: Connection timed out gpg: no valid OpenPGP data found. Error: Process completed with exit code 2.

@rbhanda Could you let me know when this issue will get fixed, Below is the script i am using to download the package

    curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
    sudo mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg
    sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-$(lsb_release -cs)-prod $(lsb_release -cs) main" > /etc/apt/sources.list.d/dotnetdev.list'
    sudo apt-get update
    sudo apt-get install azure-functions-core-tools-3

Jamaxack commented 3 years ago

Just making some noise here 😄

hasanmanzak commented 3 years ago

Seems to be fixed..

jostFT commented 3 years ago

I can confirm that it is fixed now 👍 ❤️

GiacomoNasiVarGroup commented 3 years ago

.deb packages available again! EDIT: I'm talking about the ones you need to install mssql server odbc drivers

rafaelcaviquioli commented 3 years ago

Finally! Everybody on their way doing builds and deployments :)

tmikaeld commented 3 years ago

I'm sure there'll be no congestions when hundreds or maybe thousands of machines pull large packages at the exact same time 😄

aashu-kumari-by commented 3 years ago

Now i am getting this error - Get:21 https://releases.jfrog.io/artifactory/jfrog-debs xenial/contrib amd64 Packages [3634 B] Ign:22 https://packages.microsoft.com/ubuntu/20.04/prod focal InRelease Err:23 https://packages.microsoft.com/ubuntu/20.04/prod focal Release Could not connect to packages.microsoft.com:443 (40.74.238.15), connection timed out [IP: 40.74.238.15 443] Reading package lists... E: The repository 'https://packages.microsoft.com/ubuntu/20.04/prod focal Release' no longer has a Release file.

khandelwalankit04 commented 3 years ago

Still seeing an issue : The repository 'https://packages.microsoft.com/ubuntu/18.04/prod bionic Release' does not have a Release file.

kilobyte commented 3 years ago

Well, the speed was 58KB/s, now it's nown to 22KB/s as more machines noticed the repos are up. But I hope a few more acoustic modems will be added to the cloud. ☺

(That's in jest — thanks for making it work again!)

rbhanda commented 3 years ago

Hello, As per our infra team, you may see degraded performance but the packages should be available as of now. The original cause of this issue was because there was a regression in APT Linux repos during some feature migration work that resulted in the APT repo packages becoming unavailable on the mirrors.

Moskovych commented 3 years ago

Does anybody have some mirrors?

kilasuit commented 3 years ago

Does anybody have some mirrors?

In my bathroom, yes. For this, unfortunately not 🙁

kilobyte commented 3 years ago

apt install apt-cacher-ng then point all machines on your site to it. Should help if you download the same package multiple times; and it'll be good to leave the cache running even after the current performance degradation is resolved.

dotnet / core

Ubuntu repositories broken #6381