oz fails direct boot because requests library automatically decodes gzip content

[dustymabe]

We recently switched to the requests library in oz. This works fine for the most part but the requests library has an interesting (and mostly useful) feature where The gzip and deflate transfer-encodings are automatically decoded for you.

This does not play well with oz when trying to calculate if the file received was the right content length. We write out the response from the requests library to a file and then we check that the file size matches the content length from the http header. If it doesn't match, we move on to trying to boot from a boot.iso, which also doesn't work for aarch64.

You can see this in action here:

In [1]: import requests
@
In [2]: import logging
@
In [3]: logging.basicConfig(level=logging.DEBUG)
@
In [4]: r = requests.get('https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170521.n.0/compose/Everything/aarch64/os/images/pxeboot/vmlinuz')
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): kojipkgs.fedoraproject.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /compose/branched/Fedora-26-20170521.n.0/compose/Everything/aarch64/os/images/pxeboot/vmlinuz HTTP/1.1" 200 6529431
@
In [5]: print r.headers
{'AppTime': 'D=157078', 'Content-Length': '6529431', 'AppServer': 'proxy01.phx2.fedoraproject.org', 'Accept-Ranges': 'bytes', 'Strict-Transport-Security': 'max-age=15768000; includeSubDomains; preload, max-age=15768000; includeSubDomains; preload', 'Keep-Alive': 'timeout=15, max=500', 'Server': 'Apache/2.4.25 (Fedora)', 'Last-Modified': 'Tue, 09 May 2017 16:00:17 GMT', 'Connection': 'Keep-Alive', 'Date': 'Mon, 22 May 2017 14:53:49 GMT', 'Content-Type': 'application/octet-stream', 'Content-Encoding': 'gzip'}
@
In [6]: print len(r.content)
15618560
@

So the Content-Length=6529431, but the actual content is decompressed and of length 15618560.

Why haven't we seen this before?

When dealing with x86_64 kernels, they don't have 'Content-Type': 'application/octet-stream', 'Content-Encoding': 'gzip' headers, which means the requests library doesn't decode them by default:

In [7]: z = requests.get('https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170521.n.0/compose/Everything/x86_64/os/images/pxeboot/vmlinuz')
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): kojipkgs.fedoraproject.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /compose/branched/Fedora-26-20170521.n.0/compose/Everything/x86_64/os/images/pxeboot/vmlinuz HTTP/1.1" 200 7282776
@
In [8]: print z.headers
{'AppTime': 'D=152859', 'Content-Length': '7282776', 'AppServer': 'proxy10.phx2.fedoraproject.org', 'Accept-Ranges': 'bytes', 'Strict-Transport-Security': 'max-age=15768000; includeSubDomains; preload, max-age=15768000; includeSubDomains; preload', 'Keep-Alive': 'timeout=15, max=500', 'Server': 'Apache/2.4.25 (Fedora)', 'Last-Modified': 'Tue, 09 May 2017 15:43:13 GMT', 'Connection': 'Keep-Alive', 'Date': 'Mon, 22 May 2017 14:56:58 GMT'}
@
In [9]: print len(z.content)
7282776
@

This is because the kernel files themselves are different formats so the server is handling them differently:

+[dustymabe@media ~]$ file x86_64-kernel 
x86_64-kernel: Linux kernel x86 boot executable bzImage, version 4.11.0-2.fc26.x86_64 (mockbuild@bkernel01.phx2.fedoraproject.org) #1 SMP Tue May 9 15:24:49 UTC, RO-rootFS, swap_dev 0x6, Normal VGA
+[dustymabe@media ~]$ file aarch64-kernel 
aarch64-kernel: gzip compressed data, max compression, from Unix

[dustymabe]

Note that we worked around this in the fedora infra side for now with this commit. See this ticket for details.

This means that the reproducer I put in the issue above won't work with those exact URLs.