403 on https://packages.microsoft.com/ubuntu/22.04/prod

[SMAVRillo]

Describe the issue Suddenly getting 403 on packages.microsoft.com within Github Actions Workflow:

Err:12 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 mssql-tools18 amd64 18.2.1.1-1 403  Forbidden [IP: *.*.*.* 443]

When did the issue occur? This step within our Github Actions Workflow installs the prerequisites we need for setting up our automated testing, we're installing mssql-tools18 to execute MSSQL commands from CLI

      - name: Install Prerequisites
        run: |
          sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
          curl -sSL -O https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb
          sudo dpkg -i packages-microsoft-prod.deb
          rm packages-microsoft-prod.deb
          sudo apt-get update && sudo apt-get install -y mssql-tools18 unixodbc-dev libonig-dev libmcrypt-dev mcrypt liblz4-dev liblzf-dev libzstd-dev gconf-service libstdc++6 ca-certificates libnss3 lsb-release wget libgbm-dev

Steps to Reproduce

Actual Result

Err:12 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 mssql-tools18 amd64 18.2.1.1-1 403  Forbidden [IP: *.*.*.* 443]

E: Failed to fetch https://pmc-geofence.trafficmanager.net/ubuntu/22.04/prod/pool/main/m/mssql-tools18/mssql-tools18_18.2.1.1-1_amd64.deb?geofence=true  403  Forbidden [IP: *.*.*.* 443]

Expected Result

mssql-tools18 successfully installs and continues with the rest of out workflow.

Screenshots

Additional context

[ferrouswheel]

We are also experiencing this in our Github Actions Workflow.

[NzKyle]

I too am experiencing these issues today.

It's happening from my local machine, GitHub Actions and ADO pipelines.

> [3/3] RUN apt-get install -y mssql-server-fts: 0.353 Reading package lists... 1.070 Building dependency tree... 1.205 Reading state information... 1.315 The following additional packages will be installed: 1.315 mssql-server 1.334 The following NEW packages will be installed: 1.334 mssql-server mssql-server-fts 1.974 0 upgraded, 2 newly installed, 0 to remove and 34 not upgraded. 1.974 Need to get 504 MB of archives. 1.974 After this operation, 1869 MB of additional disk space will be used. 1.974 Err:1 https://packages.microsoft.com/ubuntu/20.04/mssql-server-2022 focal/main amd64 mssql-server amd64 16.0.4105.2-2 1.974 403 Forbidden [IP: 4.193.246.172 443] 2.108 Err:2 https://packages.microsoft.com/ubuntu/20.04/mssql-server-2022 focal/main amd64 mssql-server-fts amd64 16.0.4105.2-2 2.108 403 Forbidden [IP: 4.193.246.172 443] 2.113 E: Failed to fetch https://pmc-geofence.trafficmanager.net/ubuntu/20.04/mssql-server-2022/pool/main/m/mssql-server/mssql-server_16.0.4105.2-2_amd64.deb?geofence=true 403 Forbidden [IP: 4.193.246.172 443] 2.113 E: Failed to fetch https://pmc-geofence.trafficmanager.net/ubuntu/20.04/mssql-server-2022/pool/main/m/mssql-server-fts/mssql-server-fts_16.0.4105.2-2_amd64.deb?geofence=true 403 Forbidden [IP: 4.193.246.172 443] 2.113 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

[jschroe212]

+1

[JohnKeippel]

Though I suspect this is all very related, don't want to just be a "+1":

Get:1 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 msodbcsql17 amd64 17.10.5.1-1 [749 kB] Err:2 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 mssql-tools amd64 17.10.1.1-1 403 Forbidden [IP: 4.155.111.140 443] Fetched 749 kB in 1s (938 kB/s) E: Failed to fetch https://pmc-geofence.trafficmanager.net/ubuntu/18.04/prod/pool/main/m/mssql-tools/mssql-tools_17.10.1.1-1_amd64.deb?geofence=true 403 Forbidden [IP: 4.155.111.140 443] E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

[phromo]

Also affects https://packages.microsoft.com/ubuntu/22.04/mssql-server-2022/pool/main/m/mssql-server/ packages.

[mbearup]

We're investigating the issue. Over the past several weeks we've updated our infrastructure, which includes a Web Application Firewall (WAF) to meet emerging security requirements. It's likely that the WAF is rejecting some requests based on its ruleset. We're looking into mitigation now.

[SpudInNZ]

Just chiming in here, this has blocked all our deployments, including production. I hope you can resolve this quickly @mbearup and please reach out to me if we can help with testing.

[mbearup]

This should now be mitigated. There was an issue with the WAF rules preventing download of these files. If anyone is still seeing this symptom let us know.

[JohnKeippel]

That resolved the issue for us. Thanks!

[dben-gaig]

@mbearup I'm still seeing this issue on https://packages.microsoft.com/config/rhel/8/mssql-server-2019

Status code: 403 for https://pmc-geofence.trafficmanager.net/rhel/8/mssql-server-2019/Packages/m/mssql-server-15.0.4355.3-4.x86_64.rpm?geofence=true

Let me know if you want me to create a new issue for this package.

[mbearup]

@dben-gaig I'm unable to repro this issue. I tried it 3 times in various locations, and it succeeded every time. I also see no 403 errors in our logs for this URL. Could you run the following command and paste the output here (sanitized of personal network details if needed)? FYI, one issue we've encountered is that some customers have local proxies; and since the mssql packages are served from a new (geo-fenced) domain name, they may get blocked by local proxies. If that's the case, it should be revealed by this curl command.

curl -v --head https://pmc-geofence.trafficmanager.net/rhel/8/mssql-server-2019/Packages/m/mssql-server-15.0.4355.3-4.x86_64.rpm?geofence=true

[dben-gaig]

Yep looks like a firewall issue on our side to the new domain, we'll update that. I'll follow up if I have any other issues with this.

Thanks @mbearup