search

microsoft / linux-package-repositories

Microsoft Packaged Linux Software (DEBs, RPMs, etc) are hosted on packages.microsoft.com (PMC) made available as native Linux repositories for use with package managers like APT, YUM, etc.
https://packages.microsoft.com
MIT License
69 stars 17 forks source link

Apt update fails due to "bad header data"; able to wget without issue #160

Closed jameswilddev closed 2 months ago

jameswilddev commented 2 months ago

Describe the issue

I have executed the following to try to add Microsoft's package repository as a source:

wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > packages.microsoft.gpg
sudo install -D -o root -g root -m 644 packages.microsoft.gpg /etc/apt/keyrings/packages.microsoft.gpg
curl https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/prod.list | sudo tee /etc/apt/sources.list.d/mssql-release.list

When running sudo apt update, I see the following:

Ign:5 https://packages.microsoft.com/ubuntu/24.04/prod noble InRelease
Err:5 https://packages.microsoft.com/ubuntu/24.04/prod noble InRelease
  Bad header line Bad header data [IP: 13.107.246.59 443]
Reading package lists... Done
W: Failed to fetch https://packages.microsoft.com/ubuntu/24.04/prod/dists/noble/InRelease  Bad header line Bad header data [IP: 13.107.246.59 443]
W: Some index files failed to download. They have been ignored, or old ones used instead.

I am, however, able to wget the URL without any issue:

--2024-08-09 13:54:08--  https://packages.microsoft.com/ubuntu/24.04/prod/dists/noble/InRelease
Resolving packages.microsoft.com (packages.microsoft.com)... 13.107.246.59, 2620:1ec:bdf::59
Connecting to packages.microsoft.com (packages.microsoft.com)|13.107.246.59|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3600 (3.5K) [application/octet-stream]
Saving to: ‘InRelease’

InRelease                           100%[==================================================================>]   3.52K  --.-KB/s    in 0s      

2024-08-09 13:54:08 (194 MB/s) - ‘InRelease’ saved [3600/3600]

USER@SERVER:~$ cat InRelease
-----BEGIN PGP SIGNED MESSAGE-----
(snip)

This implies that connectivity between the server and Microsoft is fine, perhaps the Microsoft end is checking the user agent/some other headers or something and returning a bad response only when apt tries to contact it?

When did the issue occur?

2024-08-09T13:58:50

If applicable, what package did you attempt to install, and from which repo?

Didn't get far enough to attempt to install any packages.

Steps to Reproduce

As above.

Actual Result

As above.

Expected Result

sudo apt update runs without error.

Screenshots

None applicable.

Additional context

Server is based out of Singapore.

LSB release:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04 LTS
Release:    24.04
Codename:   noble
jameswilddev commented 2 months ago

Seems to have been a bad public key, use the following instead of what is in the docs, then sudo apt clean

curl -fsSL https://packages.microsoft.com/keys/microsoft.asc | sudo gpg --dearmor -o /usr/share/keyrings/microsoft-prod.gpg
mbearup commented 2 months ago

The new .list files for Debian/Ubuntu repos use the signed-by directive, which expects the key to be located at /usr/share/keyrings/microsoft-prod.gpg. It looks like your initial attempt failed because the key was downloaded to a different location.

julian-klode commented 2 months ago

This is still a problem and not related to keys, it's quite simply that the server does not include any HTTP headers in the response, as you can see if you run with -o debug::Acquire::https=1:

GET /repos/code/dists/stable/InRelease HTTP/1.1
Host: packages.microsoft.com
Cache-Control: max-age=0
Accept: text/*
Range: bytes=3590-
If-Range: Mon, 19 Aug 2024 09:21:38 GMT
User-Agent: Debian APT-HTTP/1.3 (2.9.7)

Answer for: https://packages.microsoft.com/repos/code/dists/stable/InRelease
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Normally you get HTTP headers, e.g. Launchpad:

GET /deity/sid/ubuntu/dists/oracular/InRelease HTTP/1.1
Host: ppa.launchpadcontent.net
Cache-Control: max-age=0
Accept: text/*
If-Modified-Since: Thu, 15 Aug 2024 21:09:09 GMT
User-Agent: Debian APT-HTTP/1.3 (2.9.7)

Answer for: https://ppa.launchpadcontent.net/deity/sid/ubuntu/dists/oracular/InRelease
HTTP/1.1 304 Not Modified
date: Mon, 19 Aug 2024 11:08:41 GMT
server: Apache/2.4.41 (Ubuntu)
etag: "5e24-61fbf40f23c07"
expires: Mon, 19 Aug 2024 11:08:41 GMT
cache-control: max-age=0, s-maxage=270, proxy-revalidate

You can easily verify this yourself by connecting with openssl s_client and piping in the GET request.

julian-klode commented 2 months ago

Well ok openssl s_client fails with

RENEGOTIATING ERROR 80A6D8FCA1760000:error:0A00010A:SSL routines:can_renegotiate:wrong ssl version:../ssl/ssl_lib.c:2813:

julian-klode commented 2 months ago

gnutls-cli reproducer:

$ gnutls-cli packages.microsoft.com:443                                                                                                                                                                                   
Processed 146 CA certificate(s).                                                                                                                                                                                                                                                            
Resolving 'packages.microsoft.com:443'...                                                                                                                                                                                                                                                   
Connecting to '2620:1ec:bdf::45:443'...                                                                                                                                                                                                                                                     
- Certificate type: X.509                                                                                                                                                                                                                                                                   
- Got a certificate list of 3 certificates.                                                                                                                                                                                                                                                 
- Certificate[0] info:                                                                                                                                                                                                                                                                      
 - subject `CN=packages.microsoft.com,O=Microsoft Corporation,L=Redmond,ST=WA,C=US', issuer `CN=Microsoft Azure RSA TLS Issuing CA 03,O=Microsoft Corporation,C=US', serial 0x33008d2e36417315dcd74abf750000008d2e36, RSA key 4096 bits, signed using RSA-SHA384, activated `2024-08-09 18:3
8:34 UTC', expires `2025-08-04 18:38:34 UTC', pin-sha256="RWSNl6BlKSQqSQrSfj2ehJ8UTJ2X9WKkKtruu4KzsJ0="                                                                                                                                                                                     
        Public Key ID:                                                                                                                                                                                                                                                                      
                sha1:c99267ef045282d723b06489fed5ba2f938f528a                                                                                                                                                                                                                               
                sha256:45648d97a06529242a490ad27e3d9e849f144c9d97f562a42adaeebb82b3b09d
        Public Key PIN:
                pin-sha256:RWSNl6BlKSQqSQrSfj2ehJ8UTJ2X9WKkKtruu4KzsJ0= 

- Certificate[1] info:
 - subject `CN=Microsoft Azure RSA TLS Issuing CA 03,O=Microsoft Corporation,C=US', issuer `CN=DigiCert Global Root G2,OU=www.digicert.com,O=DigiCert Inc,C=US', serial 0x05196526449a5e3d1a38748f5dcfebcc, RSA key 4096 bits, signed using RSA-SHA384, activated `2023-06-08 00:00:00 UTC', expires `2026-08-25 23:59:59 UTC', pin-sha256="ZkWBotC4nL+Ba/kXaVPx7TpoRSF9uwxEAuufz67J7sQ="
- Certificate[2] info:
 - subject `CN=DigiCert Global Root G2,OU=www.digicert.com,O=DigiCert Inc,C=US', issuer `CN=DigiCert Global Root G2,OU=www.digicert.com,O=DigiCert Inc,C=US', serial 0x033af1e6a711a9a0bb2864b11d09fae5, RSA key 2048 bits, signed using RSA-SHA256, activated `2013-08-01 12:00:00 UTC', expires `2038-01-15 12:00:00 UTC', pin-sha256="i7WTqTvh0OioIruIfFR4kMPnBqrS2rdiVPl/s2uC/CY="
- Status: The certificate is trusted. 
- Description: (TLS1.3-X.509)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
- Session ID: E6:09:11:D0:99:F2:74:4E:0F:ED:7A:3A:9E:C6:5B:E9:48:8F:6F:C2:2E:6B:20:32:27:12:EB:8E:A3:2F:E7:20
- Options: OCSP status request,
- Handshake was completed

- Simple Client Mode:

GET /repos/code/dists/stable/InRelease HTTP/1.1
Host: packages.microsoft.com
Cache-Control: max-age=0
Accept: text/*
Range: bytes=3590-
If-Range: Mon, 19 Aug 2024 09:21:38 GMT
User-Agent: Debian APT-HTTP/1.3 (2.9.7)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
julian-klode commented 2 months ago

Note this only happens when we specify a range request that starts after the end of file (3590 is the size of the file). i.e. we have previously downloaded a partial Release file that is actually complete (but did not finish verification), we try to resume the download of it, and then the server responds without headers.

No range request, you get Content-Length: 3590.

If you ask it for Range: bytes=3589- it correctly returns Content-Length: 1 response with all the headers.

If you ask it for Range: bytes=3590- it returns the entire file and no headers

@jameswilddev probably ran apt clean or something in between which removed the file from /var/lib/apt/lists/partial, after which the update works again, and that sure creates some confusion :D