Open twopir opened 8 years ago
Yeah, I could validate it here... I'll try to fix it
I think that fix is still vulnerable to a problem where the gzip content is incompletely downloaded, but still expands to greater than the content-length. I don't have a demonstration handy, though.
hmm yeah, it could still be trouble... I'll try a better fix
To me, it seems like Response.raw
should be used; otherwise HTTPResponse.iter_body
will return gzip -d
'd chunks.
Greetings from 2023. This is still an issue.
[I] root@xps /tmp
# http -d 'https://XXX.blob.core.windows.net/CONTAINER/path/to/processed.json?sp=r&st=2023-10-18T11:45:05Z&se=2023-10-18T19:45:05Z&spr=https&sv=2022-11-02&sr=b&sig=PKZTjGYM0wfeU%2BpF%2FMZ93Y2RxDyVHs1xbOeg4of2nuA%3D'
HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Content-Encoding: gzip
Content-Length: 2087
Content-MD5: Q6dkiGs7Jcp6PimVCRH09A==
Content-Type: application/json
Date: Wed, 18 Oct 2023 11:45:21 GMT
ETag: "0x8DB4C84489078F7"
Last-Modified: Thu, 04 May 2023 09:45:23 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-blob-type: BlockBlob
x-ms-creation-time: Thu, 04 May 2023 09:45:23 GMT
x-ms-lease-state: available
x-ms-lease-status: unlocked
x-ms-request-id: ce24f842-101e-006a-72b8-01eea9000000
x-ms-server-encrypted: true
x-ms-version: 2022-11-02
Downloading to processed.json
Done. 4.3 kB in 00:0.10676 (39.9 kB/s)
http: LogLevel.ERROR: Incomplete download: size=2087; downloaded=4264
[I] root@xps /tmp [1]
#
To me, it seems like
Response.raw
should be used; otherwiseHTTPResponse.iter_body
will returngzip -d
'd chunks.
I think the output should be the decompressed output as it is now. It is just the warning that should go away. If the raw download matches what the http header said, all is good.
Make raw an opt-in for those cases when you really do not want the raw content encoding to be downloaded as-is. I think this issue should focus on getting rid of the error message by measuring the right thing.
It will be fixed in https://github.com/httpie/cli/pull/1531
When using HTTPie to download a file which is both gzip'd and has a content-length header sent, the final output report says that the download is incomplete. This is apparently because HTTPie tracks the number of bytes it wrote, not the number of bytes it received.
Here is an example, which reproduces in HTTPie head as of about 10 minutes ago.
Note that
size
is the value ofContent-length
, butdownloaded
is greater thansize
because of the gzip encoding.