apt fails to install from pmc-geofence.trafficmanager.net: 403 Forbidden

[skylord123]

Please check the FAQ (frequently-asked questions) first. If you have other questions or something to report, please address the following (skipping questions might delay our responses):

PHP version 8.1-fpm-bullseye (tag from the official PHP docker repo)

PHP SQLSRV or PDO_SQLSRV version
Trying to install 17

Microsoft ODBC Driver version
Not sure, fails to install.

SQL Server version
irrelevant

Client operating system
Windows running WSL with docker desktop

Table schema
irrelevant

Problem description
My build is failing when it used to work perfectly.

#0 20.26 E: Failed to fetch https://pmc-geofence.trafficmanager.net/debian/11/prod/pool/main/m/mssql-tools/mssql-tools_17.10.1.1-1_amd64.deb?geofence=true  403  Forbidden [IP: 4.155.111.140 443]
#0 20.26 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

I am unsure why it's trying to reach out to this domain when the packages exist on the microsoft website. This looks to be some sort of region redirection service that is just returning 403 Forbidden for every URL: https://pmc-geofence.trafficmanager.net/

Expected behavior and actual behavior
It should install my packages from the repo.

Repro code or steps to reproduce

This Dockerfile fails:

FROM php:8.1-fpm-bullseye

RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone \
    && apt-get update \
    && mkdir -p /etc/apt/keyrings \
    && apt-get install --no-install-recommends -y lsb-release gnupg gosu curl \
      zip unzip wget nano unixodbc unixodbc-dev \
    # add mssql repo
    && curl https://packages.microsoft.com/keys/microsoft.asc | tee /etc/apt/trusted.gpg.d/microsoft.asc \
    && bash -c "curl https://packages.microsoft.com/config/$(lsb_release -i | cut -f 2 | tr '[:upper:]' '[:lower:]')/$(lsb_release -r | cut -f 2)/prod.list > /etc/apt/sources.list.d/mssql-release.list" \
    # install packages from apt
    # mysql-community-client replaces mysql-client
    && apt-get update \
    # removed for space: build-essential software-properties-common
    && ACCEPT_EULA=Y apt-get install --no-install-recommends -y ca-certificates msodbcsql17 mssql-tools

And running that you end up getting:

#0 10.72 After this operation, 0 B of additional disk space will be used.
#0 10.72 Get:1 https://packages.microsoft.com/debian/11/prod bullseye/main amd64 msodbcsql17 amd64 17.10.5.1-1 [749 k
B]
#0 11.21 Err:2 https://packages.microsoft.com/debian/11/prod bullseye/main amd64 mssql-tools amd64 17.10.1.1-1
#0 11.21   403  Forbidden [IP: 20.242.16.222 443]
#0 11.22 Fetched 749 kB in 1s (1082 kB/s)
#0 11.22 E: Failed to fetch https://pmc-geofence.trafficmanager.net/debian/11/prod/pool/main/m/mssql-tools/mssql-tool
s_17.10.1.1-1_amd64.deb?geofence=true  403  Forbidden [IP: 20.242.16.222 443]
#0 11.22 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
------
failed to solve: executor failed running [/bin/sh -c ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone     && apt-get update     && mkdir -p /etc/apt/keyrings     && apt-get install --no-install-recommends -y
 lsb-release gnupg gosu curl       zip unzip wget nano unixodbc unixodbc-dev     && set -eux;            key='BCA4 3417 C3B4 85DD 128E C6D4 B7B3 B788 A8D3 785C';            apt-key adv --keyserver keyserver.ubuntu.com --recv-keys "$k
ey"     && echo "deb http://repo.mysql.com/apt/$(lsb_release -i | cut -f 2 | tr '[:upper:]' '[:lower:]')/ $(lsb_release -c | cut -f 2) mysql-8.0" > /etc/apt/sources.list.d/mysql.list     && curl https://packages.microsoft.com/keys/mi
crosoft.asc | tee /etc/apt/trusted.gpg.d/microsoft.asc     && bash -c "curl https://packages.microsoft.com/config/$(lsb_release -i | cut -f 2 | tr '[:upper:]' '[:lower:]')/$(lsb_release -r | cut -f 2)/prod.list > /etc/apt/sources.lis
t.d/mssql-release.list"     && apt-get update     && ACCEPT_EULA=Y apt-get install --no-install-recommends -y ca-certificates msodbcsql18 mssql-tools]: exit code: 100

Is the microsoft repo broken?

[v-maximko]

Should be fixed now, can you try again?

[skylord123]

Yes this is working now. Do we know what happened?

[v-maximko]

There was a temporary problem with the repo that has now been resolved

[jimpatrickong]

I'm getting this same error today..how do I know if there is an issue with the package?

[hannesrd]

We are getting the same error in kubernetes again. https://pmc-geofence.trafficmanager.net/debian/11/prod/pool/main/m/mssql-tools18/mssql-tools18_18.2.1.1-1_amd64.deb?geofence=true

curl also fails

[wes614]

@v-maximko just seeing if this issue was in fact now occurring again? or if we have some other issue unrelated. Seems like it could be though given other comments above.

E: Failed to fetch https://pmc-geofence.trafficmanager.net/ubuntu/16.04/prod/pool/main/m/mssql-tools/mssql-tools_17.8.1.1-1_amd64.deb?geofence=true Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification.

[skylord123]

@v-maximko just seeing if this issue was in fact now occurring again? or if we have some other issue unrelated. Seems like it could be though given other comments above.

From my experience and reading up from others: Microsoft struggles to keep their apt repo stable and online. There are numerous posts of people's CI/CD builds failing due to this issue.

Not sure if this is malicious or just gross incompetence but I find it hard to believe that a company with over 200 datacenters and a net worth exceeding 3 Trillion dollars would struggle this bad for any other reason.

[v-makouz]

@skylord123 @wes614 I tested these packages and was able to curl them without a problem, so not sure if it was intermittent or fixed already. If you're still getting the certificate error, can you try running openssl s_client -connect pmc-geofence.trafficmanager.net:443 and see if it shows Certificate chain?

We don't usually monitor closed issues, so even if it's a similar or same error as before, it's better to open a new one.

[wes614]

Thanks @v-makouz. It seems we've actually managed to get around this with the issue being related to our company's internal network security setup after we compared an openssl check from a few different machines. Appreciate the hustle by all :)

[hannesrd]

My mentioned problem was MTU. For more details and see my bug here: https://github.com/microsoft/linux-package-repositories/issues/127#issuecomment-2060426526

[trachakonda]

we're seeing this problem again, how should we get around this?