search

httpie / cli

🥧 HTTPie CLI — modern, user-friendly command-line HTTP client for the API era. JSON support, colors, sessions, downloads, plugins & more.
https://httpie.io
BSD 3-Clause "New" or "Revised" License
32.67k stars 3.68k forks source link

httpie -d does not work with gzip compressed content #1554

Open hholst80 opened 5 months ago

hholst80 commented 5 months ago

Checklist

Minimal reproduction code and steps

Expose an API endpoint that respond with gzip compressed json such as an Azure storage account.

  1. http get -d -o foo.json https://.../path/to/endpoint

Current result

+ http -d -I --check-status GET https://XXXX.blob.core.windows.net/invoices/p32132/1201394/input.json?sv=2021-12-02&se=2024-02-25T19%3A08%3A46Z&sr=c&sp=r&sig=XXXXX -o p32132/1201394/input.json
HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: x-ms-request-id,Server,x-ms-version,x-ms-resource-type,Content-Type,Content-Encoding,Last-Modified,ETag,x-ms-creation-time,Content-MD5,x-ms-lease-status,x-ms-lease-state,x-ms-blob-type,x-ms-server-encrypted,Accept-Ranges,x-ms-owner,x-ms-group,x-ms-permissions,Content-Length,Date,Transfer-Encoding
Content-Encoding: gzip
Content-Length: 55119
Content-MD5: IJ8A578M7DLY5/jW+7umKw==
Content-Type: application/json
Date: Fri, 26 Jan 2024 19:08:46 GMT
ETag: "0x8DBFA81678EA69D"
Last-Modified: Mon, 11 Dec 2023 19:43:09 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-blob-type: BlockBlob
x-ms-creation-time: Mon, 11 Dec 2023 19:36:34 GMT
x-ms-group: $superuser
x-ms-lease-state: available
x-ms-lease-status: unlocked
x-ms-owner: $superuser
x-ms-permissions: rw-r-----
x-ms-request-id: c3159c91-001e-0064-578b-5091a4000000
x-ms-resource-type: file
x-ms-server-encrypted: true
x-ms-version: 2021-12-02

Downloading to p32132/1201394/input.json
Done. 118.9 kB in 00:0.08901 (1.3 MB/s)

http: LogLevel.ERROR: Incomplete download: size=55119; downloaded=118911

…

Expected result

Httpie should consider the compressed size to validate that the entire content was downloaded, not consider the uncompressed size with the compressed gzip payload that was sent.

…

Debug output

Please re-run the command with --debug, then copy the entire command & output and paste both below:

+ http -d --debug -I --check-status GET https://XXXX.blob.core.windows.net/invoices/p32132/1201394/input.json?sv=2021-12-02&se=2024-02-25T19%3A10%3A41Z&sr=c&sp=r&sig=XXXX -o p32132/1201394/input.json
HTTPie 3.2.1
Requests 2.31.0
Pygments 2.17.2
Python 3.11.6 (main, Nov 14 2023, 18:04:26) [GCC 13.2.1 20230801]
/usr/bin/python3
Linux 6.7.0-zen3-1-zen

<Environment {'apply_warnings_filter': <function Environment.apply_warnings_filter at 0x78780a9fd120>,
 'args': Namespace(),
 'as_silent': <function Environment.as_silent at 0x78780a9fcfe0>,
 'colors': 256,
 'config': {'default_options': []},
 'config_dir': PosixPath('/home/root/.config/httpie'),
 'devnull': <property object at 0x78780a9ed4e0>,
 'is_windows': False,
 'log_error': <function Environment.log_error at 0x78780a9fd080>,
 'program_name': 'http',
 'quiet': 0,
 'rich_console': <functools.cached_property object at 0x78780a9f3050>,
 'rich_error_console': <functools.cached_property object at 0x78780a9f3150>,
 'show_displays': True,
 'stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,
 'stderr_isatty': True,
 'stdin': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>,
 'stdin_encoding': 'utf-8',
 'stdin_isatty': False,
 'stdout': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>,
 'stdout_encoding': 'utf-8',
 'stdout_isatty': True}>

<PluginManager {'adapters': [],
 'auth': [<class 'httpie.plugins.builtin.BasicAuthPlugin'>,
          <class 'httpie.plugins.builtin.DigestAuthPlugin'>,
          <class 'httpie.plugins.builtin.BearerAuthPlugin'>],
 'converters': [],
 'formatters': [<class 'httpie.output.formatters.headers.HeadersFormatter'>,
                <class 'httpie.output.formatters.json.JSONFormatter'>,
                <class 'httpie.output.formatters.xml.XMLFormatter'>,
                <class 'httpie.output.formatters.colors.ColorFormatter'>]}>

>>> requests.request(**{'auth': None,
 'data': RequestJSONDataDict(),
 'headers': <HTTPHeadersDict('User-Agent': b'HTTPie/3.2.1', 'Accept-Encoding': b'identity')>,
 'method': 'get',
 'params': <generator object MultiValueOrderedDict.items at 0x78780bca6a40>,
 'url': 'https://XXXX.blob.core.windows.net/invoices/p32132/1201394/input.json?sv=2021-12-02&se=2024-02-25T19%3A10%3A41Z&sr=c&sp=r&sig=XXXX'})

HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: x-ms-request-id,Server,x-ms-version,x-ms-resource-type,Content-Type,Content-Encoding,Last-Modified,ETag,x-ms-creation-time,Content-MD5,x-ms-lease-status,x-ms-lease-state,x-ms-blob-type,x-ms-server-encrypted,Accept-Ranges,x-ms-owner,x-ms-group,x-ms-permissions,Content-Length,Date,Transfer-Encoding
Content-Encoding: gzip
Content-Length: 55119
Content-MD5: IJ8A578M7DLY5/jW+7umKw==
Content-Type: application/json
Date: Fri, 26 Jan 2024 19:10:41 GMT
ETag: "0x8DBFA81678EA69D"
Last-Modified: Mon, 11 Dec 2023 19:43:09 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-blob-type: BlockBlob
x-ms-creation-time: Mon, 11 Dec 2023 19:36:34 GMT
x-ms-group: $superuser
x-ms-lease-state: available
x-ms-lease-status: unlocked
x-ms-owner: $superuser
x-ms-permissions: rw-r-----
x-ms-request-id: c8568e21-501e-0079-548b-509c18000000
x-ms-resource-type: file
x-ms-server-encrypted: true
x-ms-version: 2021-12-02

Downloading to p32132/1201394/input.json
Done. 118.9 kB in 00:0.08444 (1.4 MB/s)

http: LogLevel.ERROR: Incomplete download: size=55119; downloaded=118911

Additional information, screenshots, or code examples

…

Ousret commented 1 month ago

It will be fixed in https://github.com/httpie/cli/pull/1531