When getting API fee-estimates via wget or curl, sometimes get binary junk instead of json

[nanotube]

As per title, when fetching the fee-estimates API from https://blockstream.info/api/fee-estimates, using wget or curl (but not browser), it fails at apparently random times, producing binary gibberish instead of json output. See below for example:

nanotube@gribble:~$ wget -O - https://blockstream.info/api/fee-estimates --2020-05-07 19:50:28-- https://blockstream.info/api/fee-estimates Resolving blockstream.info (blockstream.info)... 35.201.74.156 Connecting to blockstream.info (blockstream.info)|35.201.74.156|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 325 [application/json] Saving to: `STDOUT'

0% [ ] 0 --.-K/s {"12":56.102,"8":58.984,"2":75.447,"5":65.449,"11":58.84,"1":75.447,"9":58.84,"4":68.081,"13":43.924,"17":43.924,"25":41.793,"6":65.449,"14":43.924,"144":6.012,"504":6.012,"3":71.51,"20":41.793,"21":41.793,"19":43.924,"16":43.924,"10":58.84,"7":65.449,"15":43.924,"18":43.924,"22":41.793,"24":41100%[=========================================================================================================>] 325 --.-K/s in 0s

2020-05-07 19:50:28 (8.15 MB/s) - written to stdout [325/325]

nanotube@gribble:~$ wget -O - https://blockstream.info/api/fee-estimates --2020-05-07 19:50:30-- https://blockstream.info/api/fee-estimates Resolving blockstream.info (blockstream.info)... 35.201.74.156 Connecting to blockstream.info (blockstream.info)|35.201.74.156|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 156 [application/json] Saving to: `STDOUT'

0% [ ] 0 --.-K/s M�A���@T�k���6�U����,>�F�uV�B�<� a:��B�V�8 ���, ��<CG5XB�^iy�i�7p�<f-���X=uO�B��� ��ɅL6P�r(���>.Ko��kRe+�͆�7����y�O]E��J>_� 100%[=========================================================================================================>] 156 --.-K/s in 0s

2020-05-07 19:50:30 (111 MB/s) - written to stdout [156/156]

nanotube@gribble:~$

Steps to reproduce: try curl-ing or wget-ing the api point a few times.

[shesek]

This is happening because of gzip encoding. This should work: curl --compressed https://blockstream.info/api/fee-estimates

The odd thing is that for some reason, some responses are gzipped while others aren't. I'm not quite sure why...

Edit: Maybe there's some minimum size for enabling gzip, which is sometimes met and sometimes not depending on the exact estimates being returned?

[nanotube]

Seems like a bug that needs to be fixed on the esplora side, then? since API behavior should be consistent...

[shesek]

I would not consider this a bug, its a valid HTTP response that HTTP clients should be able to understand.

I'm not sure if the size of the response is really the cause here, but if so then it makes sense to avoid gzipping small responses. nginx has a configuration option for that called gzip_min_length.

[nanotube]

Thank you for your quick response.

I thought if the client doesn't ask for compression via the Accept-Encoding header, the server is not supposed to randomly decide to compress things?

To clarify: I am not running my own instance of esplora, I'm accessing the one on blockstream.info, and thought this was some bug in the esplora code.

But now I see it is likely a problem in the webserver configuration on blockstream.info... Is there a good place to report that? Or do you not think it is a problem and I should just suck it up and make my python code check for gzip encoding?

[shesek]

I thought if the client doesn't ask for compression via the Accept-Encoding header, the server is not supposed to randomly decide to compress things?

Actually, yes, you're right - it is a bug if gzip isn't being asked for by the client.

This is the right place to report, but we should ping @greenaddress for this as I'm not involved with the devops aspect of esplora (the gzip encoding is added by some reverse proxy standing in the middle, not by esplora itself).

should just suck it up and make my python code check for gzip encoding?

I think I would probably just find an http library that supported gzip encoding out of the box :roll_eyes:

[nanotube]

The built-in urllib/Request stuff supports it just fine, just need to do a simple conditional.

Just never had to do that before since this is the first time i run into a server that assumes I want gzip without the accept-encoding. Now that the appropriate people have been notified, perhaps i'll do that anyway.

Not sure what kind of proxying/caching is happening, perhaps it just needs to add a variance for accept-encoding in the cache.

thank you again for your help and responses!