search

jawah / niquests

“Safest, Fastest, Easiest, and Most advanced” Python HTTP Client. Production Ready! Drop-in replacement for Requests. HTTP/1.1, HTTP/2, and HTTP/3 supported.
https://niquests.readthedocs.io/en/latest/
Apache License 2.0
790 stars 19 forks source link

Unable to get http3 request against pie.dev #60

Closed dwt closed 8 months ago

dwt commented 8 months ago

Hi there, I was directed here by @Ousret from a merge request on httpie - hopefully this is the right location to file this bug. Please note, I would be happy to provide any additional information and try out experiments if that helps debug this issue.

Summary

I expected to get a http3 connection to pie.dev

❯ python
Python 3.12.1 (main, Dec  8 2023, 18:57:37) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from niquests import Session
>>> 
>>> with Session() as s:
...     print(s.get("https://pie.dev/get"))
... 
<Response HTTP/2 [200]>
>>> from urllib3.contrib.hface import HTTPProtocolFactory, HTTP3Protocol
>>> 
>>> print(HTTPProtocolFactory.new(HTTP3Protocol))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dwt/.virtualenvs/tempenv-714d1721928c9/lib/python3.12/site-packages/urllib3/contrib/hface/protocols/_factories.py", line 126, in new
    return implementation_target(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: HTTP3ProtocolAioQuicImpl.__init__() missing 3 required keyword-only arguments: 'remote_address', 'server_name', and 'tls_config'

Reproduction Steps

See above

System Information

$ python -m niquests.help
❯ python --version
Python 3.12.1

~/C/P/httpie  🐍 tempenv-714d1721928c9 🌱 feature-tryout-niquests  
❯ macosver
10.16

~/C/P/httpie  🐍 tempenv-714d1721928c9 🌱 feature-tryout-niquests  
❯ pip list
Package            Version    Editable project location
------------------ ---------- -------------------------------
certifi            2023.11.17
cffi               1.16.0
charset-normalizer 3.3.2
cryptography       41.0.7
defusedxml         0.7.1
h11                0.14.0
h2                 4.1.0
hpack              4.0.0
httpie             4.0.0b1    /Users/dwt/Code/Projekte/httpie
hyperframe         6.0.1
idna               3.6
kiss-headers       2.4.3
markdown-it-py     3.0.0
mdurl              0.1.2
multidict          6.0.4
niquests           3.4.0
pip                23.3.2
pycparser          2.21
Pygments           2.17.2
python-socks       2.4.4
qh3                0.14.0
requests           2.31.0
requests-toolbelt  1.0.0
rich               13.7.0
setuptools         69.0.3
urllib3            2.1.0
urllib3-future     2.4.902
wassima            1.0.3

~/C/P/httpie  🐍 tempenv-714d1721928c9 🌱 feature-tryout-niquests  
❯ python -m niquests.help
{
  "charset_normalizer": {
    "version": "3.3.2"
  },
  "cryptography": {
    "version": "41.0.7"
  },
  "http1": {
    "h11": "0.14.0"
  },
  "http2": {
    "h2": "4.1.0"
  },
  "http3": {
    "enabled": true,
    "qh3": "0.14.0"
  },
  "idna": {
    "version": "3.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.12.1"
  },
  "niquests": {
    "version": "3.4.0"
  },
  "ocsp": {
    "enabled": true
  },
  "platform": {
    "release": "22.6.0",
    "system": "Darwin"
  },
  "system_ssl": {
    "version": "30200000"
  },
  "urllib3.future": {
    "version": "2.4.902"
  },
  "wassima": {
    "certifi_fallback": false,
    "enabled": true,
    "version": "1.0.3"
  }
}
Ousret commented 8 months ago

OK, let's try to get more intel.

import logging
from niquests import Session

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
explain_handler = logging.StreamHandler()
explain_handler.setFormatter(
    logging.Formatter("%(asctime)s | %(levelname)s | %(message)s")
)
logger.addHandler(explain_handler)

with Session() as s:
     print(s.get("https://pie.dev/get"))
dwt commented 8 months ago 
❯ python
Python 3.12.1 (main, Dec  8 2023, 18:57:37) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import logging
>>> from niquests import Session
>>> 
>>> logger = logging.getLogger()
>>> logger.setLevel(logging.DEBUG)
>>> explain_handler = logging.StreamHandler()
>>> explain_handler.setFormatter(
...     logging.Formatter("%(asctime)s | %(levelname)s | %(message)s")
... )
>>> logger.addHandler(explain_handler)
>>> 
>>> with Session() as s:
...      print(s.get("https://pie.dev/get"))
... 
2024-01-03 12:07:11,846 | DEBUG | Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=None, status=None)
2024-01-03 12:07:11,847 | DEBUG | Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=None, status=None)
2024-01-03 12:07:11,859 | DEBUG | Starting new HTTPS connection (1): pie.dev:443
2024-01-03 12:07:12,136 | DEBUG | Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=None, status=None)
2024-01-03 12:07:12,136 | DEBUG | Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=None, status=None)
2024-01-03 12:07:12,138 | DEBUG | Starting new HTTP connection (1): e1.o.lencr.org:80
2024-01-03 12:07:12,250 | DEBUG | http://e1.o.lencr.org:80 "POST / HTTP/1.1" 200 344
2024-01-03 12:07:12,253 | DEBUG | Adding (b':method', b'GET') to the header table, sensitive:False, huffman:True
2024-01-03 12:07:12,253 | DEBUG | Encoding 2 with 7 bits
2024-01-03 12:07:12,253 | DEBUG | Adding (b':scheme', b'https') to the header table, sensitive:False, huffman:True
2024-01-03 12:07:12,253 | DEBUG | Encoding 7 with 7 bits
2024-01-03 12:07:12,253 | DEBUG | Adding (b':path', b'/get') to the header table, sensitive:False, huffman:True
2024-01-03 12:07:12,253 | DEBUG | Encoding 4 with 6 bits
2024-01-03 12:07:12,253 | DEBUG | Encoding 3 with 7 bits
2024-01-03 12:07:12,253 | DEBUG | Adding (b':authority', b'pie.dev') to the header table, sensitive:False, huffman:True
2024-01-03 12:07:12,253 | DEBUG | Encoding 1 with 6 bits
2024-01-03 12:07:12,253 | DEBUG | Encoding 5 with 7 bits
2024-01-03 12:07:12,253 | DEBUG | Adding (b'user-agent', b'niquests/3.4.0') to the header table, sensitive:False, huffman:True
2024-01-03 12:07:12,253 | DEBUG | Encoding 58 with 6 bits
2024-01-03 12:07:12,253 | DEBUG | Encoding 10 with 7 bits
2024-01-03 12:07:12,253 | DEBUG | Adding (b'accept-encoding', b'gzip, deflate') to the header table, sensitive:False, huffman:True
2024-01-03 12:07:12,254 | DEBUG | Encoding 16 with 7 bits
2024-01-03 12:07:12,254 | DEBUG | Adding (b'accept', b'*/*') to the header table, sensitive:False, huffman:True
2024-01-03 12:07:12,254 | DEBUG | Encoding 19 with 6 bits
2024-01-03 12:07:12,254 | DEBUG | Encoding 3 with 7 bits
2024-01-03 12:07:12,254 | DEBUG | Encoded header block to b'\x82\x87D\x83bb\xa7A\x85\xac\xc5^B\xf7z\x8a\xa8\xdd\xad*\x12\x86\x19]\xa5\xc1\x90S\x83\xf9c\xe7'
2024-01-03 12:07:12,305 | DEBUG | Decoding b'?\xe1\x1f\x88a\x96\xe4Y>\x94\x03*e\x1dJ\x08\x02i@\x86\xe0\x1d\xb8\x11)\x8bF\xff_\x8b\x1du\xd0b\r&=LtA\xeaT\x01*@\x96\x19\x08T!b\x1e\xa4\xd8z\x16\x1d\x14\x1f\xc2\xc4\xb0\xb2\x16\xa4\x98t#\x83M\x96\x97@\x8a$\xab\x10d\x9c\xab!#M\xa8\x86\xbf\xcfL:2^@\x87\xb0\xb5\x9e\xc4\xac\x93\xff\xffH\xff\xfd\xfc\x96\xa9+9\xaaJ?\x9b\x9f\xfb\xff\xfd\xfc\xdbe\x1f\xcd\xcf\xe6t\xa6\xb4\\\xff\xfe\x0c\x7f\xff\x06\x06\xbdE\xa1rP{d\x96\x81\xd8U\xc8z\x7f\xff\x83\x16\x16\xb3\xd8\x9f\xff\xe0\xc7v\x7f\xc4A\x9f\x81U\x15\xd1\x8b\x898\xee\xbe\x87\xf0\xe2\xd6a\xdb\xd02*+\xbbp\xf1\xd5+\x93TuE\x86\x95Eu\x00\x9e\x91\xef\xcaX\xe6\xd7\xeb~\x9f\n\x8b\x0e\t\xaf\xedg\x9b\xdc3\xf1\xf5\x82\xee\xc0>4\xfe~\xf6j\xfe\xb9\xe3\x7f\xd8=\x14\x98a\x0c\xc3\xb3\x87z\xc8\x8f^\xf68\xe7EE\x87Sx\x80\xdf\xefLCCro4\x8e\xbe\xdf\x1f\xe7\xff\xdf\xfe}\x7f3X{k\xfen\x7f$\x95j\x8bG\xf3\xf5\xfc\xd2?1\x0eb\xff7\x1c\x03O\x00\x1f\xfe\xff@\x03nel\xb1\xff\xfd\xfc\xa2\xd2\x10\xa8DR\xd82$\xc7\xab\xf9\xb8\x0f\xaf\xe6\xc2\xd6{\x13\x12O\xfc\xdc\xfeI*\xd5\x16\x8f\xe7\xeb\xf9\xa4~b\x1c\xc5\xfen8\x06\x9e\x00?\xfdv\x87%\x07\xb6Ih\x1d\x85@\x85$\xabX?_\x8fy\x99FG$|\xa5}\xd8\xdfm\xa5\xa1\xd1\xbbZ\x83\x9b\xd9\xab@\x85\x1d\tY\x1d\xc9\x90\x9d\x98?\x9b\x8d4\xcf\xf3\xf6\xa5#\x81\xe7\x1a\x00?'
2024-01-03 12:07:12,305 | DEBUG | Decoded 4096, consumed 3 bytes
2024-01-03 12:07:12,305 | DEBUG | Resizing header table to 4096 from 4096
2024-01-03 12:07:12,305 | DEBUG | Decoded 8, consumed 1 bytes
2024-01-03 12:07:12,305 | DEBUG | Decoded (b':status', b'200'), consumed 1
2024-01-03 12:07:12,305 | DEBUG | Decoded 33, consumed 1 bytes
2024-01-03 12:07:12,305 | DEBUG | Decoded 22, consumed 1 bytes
2024-01-03 12:07:12,305 | DEBUG | Decoded (b'date', b'Wed, 03 Jan 2024 11:07:12 GMT'), total consumed 24 bytes, indexed True
2024-01-03 12:07:12,305 | DEBUG | Decoded 31, consumed 1 bytes
2024-01-03 12:07:12,305 | DEBUG | Decoded 11, consumed 1 bytes
2024-01-03 12:07:12,305 | DEBUG | Decoded (b'content-type', b'application/json'), total consumed 13 bytes, indexed True
2024-01-03 12:07:12,305 | DEBUG | Decoded 20, consumed 1 bytes
2024-01-03 12:07:12,305 | DEBUG | Decoded 1, consumed 1 bytes
2024-01-03 12:07:12,305 | DEBUG | Decoded (b'access-control-allow-origin', <memory at 0x102dfb700>), total consumed 3 bytes, indexed True
2024-01-03 12:07:12,305 | DEBUG | Decoded 22, consumed 1 bytes
2024-01-03 12:07:12,305 | DEBUG | Decoded 3, consumed 1 bytes
2024-01-03 12:07:12,305 | DEBUG | Decoded (b'access-control-allow-credentials', b'true'), total consumed 28 bytes, indexed True
2024-01-03 12:07:12,305 | DEBUG | Decoded 10, consumed 1 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded 6, consumed 1 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded (b'cf-cache-status', b'DYNAMIC'), total consumed 19 bytes, indexed True
2024-01-03 12:07:12,306 | DEBUG | Decoded 7, consumed 1 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded 199, consumed 2 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded (b'report-to', b'{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=LUe%2Ba2VcVSDs9FGPiauj1d%2BRFVOf6gno%2Fm%2Bs0hmaTJebgPyTNw%2FEgDR3Y8ULVyEBQ09atXZq4DPhb9z0yecFA1garUvpcsyzQ66j%2FO5G05ZjGas5dTid795V"}],"group":"cf-nel","max_age":604800}'), total consumed 210 bytes, indexed True
2024-01-03 12:07:12,306 | DEBUG | Decoded 3, consumed 1 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded 49, consumed 1 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded (<memory at 0x102dfb1c0>, b'{"success_fraction":0,"report_to":"cf-nel","max_age":604800}'), total consumed 55 bytes, indexed True
2024-01-03 12:07:12,306 | DEBUG | Decoded 54, consumed 1 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded 7, consumed 1 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded (b'server', b'cloudflare'), total consumed 9 bytes, indexed True
2024-01-03 12:07:12,306 | DEBUG | Decoded 5, consumed 1 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded 15, consumed 1 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded (b'cf-ray', b'83fac6d9ee97b954-AMS'), total consumed 23 bytes, indexed True
2024-01-03 12:07:12,306 | DEBUG | Decoded 26, consumed 1 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded 3, consumed 1 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded (b'content-encoding', b'gzip'), total consumed 5 bytes, indexed True
2024-01-03 12:07:12,306 | DEBUG | Decoded 5, consumed 1 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded 16, consumed 1 bytes
2024-01-03 12:07:12,306 | DEBUG | Decoded (b'alt-svc', b'h3=":443"; ma=86400'), total consumed 24 bytes, indexed True
2024-01-03 12:07:12,306 | DEBUG | https://pie.dev:443 "GET /get HTTP/2.0" 200 None
<Response HTTP/2 [200]>
>>>
dwt commented 8 months ago

I would love some info on how you read that, the logs seem to indicate (to me) that niquests starts a http 1.1 connection which is then upgraded to http2?

Ousret commented 8 months ago

Everything seems fine on Niquests side. I have a reasonable explanation onto why. Forget about the HTTP/1.1 it is a OCSP request made to ensure you're not getting MITM-hacked.

from niquests import Session

with Session() as s:
      print(s.get("https://pie.dev/get"))
      print(s.get("https://pie.dev/get"))

Then run

from niquests import Session

with Session(resolver="doh+google://") as s:
      print(s.get("https://pie.dev/get"))

On the HTTPie side, verify your config directory is actually writable.

Run (twice):

https pie.dev/get

and

https --resolver "doh+google://" pie.dev/get

You should have a file in /Users/dwt/.config/httpie named quic.json.

Finally, I did find a bug in the PR. The caching layer wasn't properly injected in the custom HTTPSAdapter, and made the --http3 flag useless.

I fixed it. Thank you for the report and debug.

Tell me if this (recent push) fixed the problem.

Ousret commented 8 months ago

Do you confirm its fixed? If so, feel free to close this issue.

Ousret commented 8 months ago

Re:

I am going to suppose everything is OK on your side. If not, do not hesitate to let me know.

Regards,

dwt commented 8 months ago

Sorry, I was a bit away the last few days - I have retested now and had these findings:

❯ python
Python 3.12.1 (main, Dec  8 2023, 18:57:37) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from niquests import Session

with Session() as s:
      print(s.get("https://pie.dev/get"))
      print(s.get("https://pie.dev/get"))

>>> 
>>> with Session() as s:
...       print(s.get("https://pie.dev/get"))
...       print(s.get("https://pie.dev/get"))
... 
<Response HTTP/2 [200]>
<Response HTTP/3 [200]>
>>> ^D

~/C/P/httpie  🐍 tempenv-714d1721928c9 🌱 feature-tryout-niquests  6s 
❯ python
Python 3.12.1 (main, Dec  8 2023, 18:57:37) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from niquests import Session

with Session(resolver="doh+google://") as s:
      print(s.get("https://pie.dev/get"))

>>> 
>>> with Session(resolver="doh+google://") as s:
...       print(s.get("https://pie.dev/get"))
... 
<Response HTTP/3 [200]>
>>> ^D

This seems to work - could you perhaps elaborate why the first request didn't work but the second did? Does the session need to collect the information that the host is http3 capable before trying it?

On the shell I was not so lucky:

❯ https pie.dev/get
HTTP/2 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Alt-Svc: h3=":443"; ma=86400
Cf-Cache-Status: DYNAMIC
Cf-Ray: 842cb3d0b8d70baa-AMS
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 09 Jan 2024 12:32:20 GMT
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=2oJxp%2FRmNZ11y7hhOr9bkcI3MRmBHh0wyDGi5ZbvJbFo7tkxRJ7ULTPR%2F4EJslQGJhR4B2jh2zwO9BkNA8Jl55t2n%2FAHrW99AC7QjOTWDWqvQhgGvSdkuZhV"}],"group":"cf-nel","max_age":604800}
Server: cloudflare

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "91.65.247.116",
        "Cf-Ipcountry": "DE",
        "Cf-Ray": "842cb3d0b8d70baa-FRA",
        "Cf-Visitor": "{\"scheme\":\"https\"}",
        "Connection": "Keep-Alive",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/4.0.0.b1"
    },
    "origin": "91.65.247.116",
    "url": "https://pie.dev/get"
}

~/C/P/httpie  🐍 tempenv-714d1721928c9 🌱 feature-tryout-niquests  
❯ https pie.dev/get
HTTP/2 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Alt-Svc: h3=":443"; ma=86400
Cf-Cache-Status: DYNAMIC
Cf-Ray: 842cb3e7be386fab-CDG
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 09 Jan 2024 12:32:24 GMT
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=x7%2BYUHaNka9LBnCLM6ndKctXAAokJ1BGvBYvJF9pdbc0yjLZowYIT68ULkJ2sQyw50STbTjTWS8uE2wWAb0%2BmMkbE4JWrCczanMqBw%2BN8Mw0EH24c7eG1RVY"}],"group":"cf-nel","max_age":604800}
Server: cloudflare

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "91.65.247.116",
        "Cf-Ipcountry": "DE",
        "Cf-Ray": "842cb3e7be386fab-FRA",
        "Cf-Visitor": "{\"scheme\":\"https\"}",
        "Connection": "Keep-Alive",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/4.0.0.b1"
    },
    "origin": "91.65.247.116",
    "url": "https://pie.dev/get"
}

~/C/P/httpie  🐍 tempenv-714d1721928c9 🌱 feature-tryout-niquests  
❯ https --resolver "doh+google://" pie.dev/get
HTTP/2 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Alt-Svc: h3=":443"; ma=86400
Cf-Cache-Status: DYNAMIC
Cf-Ray: 842cb4360fa76627-AMS
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 09 Jan 2024 12:32:36 GMT
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Fp5K55yzrboR%2FvkSr%2FPSGCQDM4Ghoo0SrBN%2B6c4aGvLtcMZ%2Byb4YFsyN5cXTPwzXXNQ4pQuOSO95TJcA%2FscaiqVFzaZhOREMhbW4n3atnpWkDsFXVNXKsK1B"}],"group":"cf-nel","max_age":604800}
Server: cloudflare

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "91.65.247.116",
        "Cf-Ipcountry": "DE",
        "Cf-Ray": "842cb4360fa76627-FRA",
        "Cf-Visitor": "{\"scheme\":\"https\"}",
        "Connection": "Keep-Alive",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/4.0.0.b1"
    },
    "origin": "91.65.247.116",
    "url": "https://pie.dev/get"
}

~/C/P/httpie  🐍 tempenv-714d1721928c9 🌱 feature-tryout-niquests  
❯ https --resolver "doh+google://" pie.dev/get
HTTP/2 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Alt-Svc: h3=":443"; ma=86400
Cf-Cache-Status: DYNAMIC
Cf-Ray: 842cb44ccf350a6f-AMS
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 09 Jan 2024 12:32:40 GMT
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=23Q18VEt4d3YuApMUBS0NTExzN7TV5dpwqZTzifTgpck9dNsMyCCsyur8MPNw%2BIJy361tQXa3hGY%2F%2FzQzq3%2Fx1kF2FooPIdblwTZQLSIeDDSxchRi6mKuwmO"}],"group":"cf-nel","max_age":604800}
Server: cloudflare

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "91.65.247.116",
        "Cf-Ipcountry": "DE",
        "Cf-Ray": "842cb44ccf350a6f-FRA",
        "Cf-Visitor": "{\"scheme\":\"https\"}",
        "Connection": "Keep-Alive",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/4.0.0.b1"
    },
    "origin": "91.65.247.116",
    "url": "https://pie.dev/get"
}

~/C/P/httpie  🐍 tempenv-714d1721928c9 🌱 feature-tryout-niquests  
❯ https --http3 --resolver "doh+google://" pie.dev/get
HTTP/2 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Alt-Svc: h3=":443"; ma=86400
Cf-Cache-Status: DYNAMIC
Cf-Ray: 842cb5b37abe6f04-CDG
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 09 Jan 2024 12:33:37 GMT
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=6R7RN875gKI2cArgg3Mw9EHgmOQdoP5FgQUhkDgs%2F4A6QPxZjnzvdei1aMvyxovWIxvVNdYjScTDrYr%2BtTnHdMKOTePN6X0O4bqBgDplqFoLr1L3%2BRV54SZp"}],"group":"cf-nel","max_age":604800}
Server: cloudflare

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "91.65.247.116",
        "Cf-Ipcountry": "DE",
        "Cf-Ray": "842cb5b37abe6f04-FRA",
        "Cf-Visitor": "{\"scheme\":\"https\"}",
        "Connection": "Keep-Alive",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/4.0.0.b1"
    },
    "origin": "91.65.247.116",
    "url": "https://pie.dev/get"
}

~/C/P/httpie  🐍 tempenv-714d1721928c9 🌱 feature-tryout-niquests  
❯ https --http3 pie.dev/get
HTTP/2 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Alt-Svc: h3=":443"; ma=86400
Cf-Cache-Status: DYNAMIC
Cf-Ray: 842cb5db7d00f110-CDG
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 09 Jan 2024 12:33:44 GMT
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Jhqw1wkYiH55ne8D4%2B4qGnyyHbbx01QNCTZcCcqQYFuXv%2FwQb3ZGu%2FYmZI6VBiglfw4GXmo2XhG0F26m9ZdSgur4U3Gl68Rz83DOglkIAgtv4NtTYxWz9aor"}],"group":"cf-nel","max_age":604800}
Server: cloudflare

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "91.65.247.116",
        "Cf-Ipcountry": "DE",
        "Cf-Ray": "842cb5db7d00f110-FRA",
        "Cf-Visitor": "{\"scheme\":\"https\"}",
        "Connection": "Keep-Alive",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/4.0.0.b1"
    },
    "origin": "91.65.247.116",
    "url": "https://pie.dev/get"
}

All the while the config directory seems to be perfectly writeable, but no config file shows up there.

~/.config/httpie 
❯ ls -al .
total 8
drwxr-xr-x   3 dwt  staff   96  3 Jan 10:43 ./
drwxr-xr-x  23 dwt  staff  736  3 Jan 10:43 ../
-rw-r--r--   1 dwt  staff  221  3 Jan 10:43 version_info.json

If you give me a point where to start, I can try to debug the library to see why it can't seem to write in that directory?

Ousret commented 8 months ago

OK. Let's try to further understand your case. Immediately I would try to remove httpie completely and re do the installation of the fork/patch and try again. I am suspicious on how --http3 did not work. I think it will resolve your case definitely. Let me know.

Ousret commented 8 months ago

could you perhaps elaborate why the first request didn't work but the second did? Does the session need to collect the information that the host is http3 capable before trying it?

without a custom DNS, you cannot reach a HTTP/3 endpoint without prior establishing a HTTP/1 or HTTP/2 link, that's why. but this https --resolver "doh+google://" pie.dev/get seems weird it did not complete to HTTP/3,

dwt commented 8 months ago

Not sure, it seems that your assesment that the quic config file is not written still seems correct.

Even when recreating the full venv and trying the requests multiple times that file is not written - and of course the requests stay at http/2

Details

```shell ❯ cd Code/Projekte/httpie/ ~/C/P/httpie 🐍 🌱 feature-tryout-niquests ❯ vf tmp Creating tempenv-51b5170562de8 via ~/.local/pipx/venvs/virtualfish/bin/python … ~/C/P/httpie 🐍 tempenv-51b5170562de8 🌱 feature-tryout-niquests 3s ❯ git pull Already up to date. ~/C/P/httpie 🐍 tempenv-51b5170562de8 🌱 feature-tryout-niquests 2s ❯ git remote -v origin git@github.com:Ousret/httpie.git (fetch) origin git@github.com:Ousret/httpie.git (push) ~/C/P/httpie 🐍 tempenv-51b5170562de8 🌱 feature-tryout-niquests ❯ pip install --editable . Obtaining file:///Users/dwt/Code/Projekte/httpie Installing build dependencies ... done Checking if build backend supports build_editable ... done Getting requirements to build editable ... done Preparing editable metadata (pyproject.toml) ... done Requirement already satisfied: pip in /Users/dwt/.virtualenvs/tempenv-51b5170562de8/lib/python3.12/site-packages (from httpie==4.0.0b1) (23.3.2) Collecting charset-normalizer>=2.0.0 (from httpie==4.0.0b1) Using cached charset_normalizer-3.3.2-cp312-cp312-macosx_11_0_arm64.whl.metadata (33 kB) Collecting defusedxml>=0.6.0 (from httpie==4.0.0b1) Using cached defusedxml-0.7.1-py2.py3-none-any.whl (25 kB) Collecting niquests<4,>=3.4.0 (from niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Downloading niquests-3.4.1-py3-none-any.whl.metadata (6.4 kB) Collecting Pygments>=2.5.2 (from httpie==4.0.0b1) Using cached pygments-2.17.2-py3-none-any.whl.metadata (2.6 kB) Collecting setuptools (from httpie==4.0.0b1) Using cached setuptools-69.0.3-py3-none-any.whl.metadata (6.3 kB) Collecting rich>=9.10.0 (from httpie==4.0.0b1) Using cached rich-13.7.0-py3-none-any.whl.metadata (18 kB) Collecting idna<4,>=2.5 (from niquests<4,>=3.4.0->niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Using cached idna-3.6-py3-none-any.whl.metadata (9.9 kB) Collecting kiss-headers<4,>=2 (from niquests<4,>=3.4.0->niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Using cached kiss_headers-2.4.3-py3-none-any.whl.metadata (13 kB) Collecting urllib3-future<3,>=2.4.901 (from niquests<4,>=3.4.0->niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Downloading urllib3_future-2.4.903-py3-none-any.whl.metadata (6.1 kB) Collecting wassima<2,>=1.0.1 (from niquests<4,>=3.4.0->niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Using cached wassima-1.0.3-cp37-abi3-macosx_11_0_arm64.whl.metadata (3.8 kB) Collecting markdown-it-py>=2.2.0 (from rich>=9.10.0->httpie==4.0.0b1) Using cached markdown_it_py-3.0.0-py3-none-any.whl.metadata (6.9 kB) Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich>=9.10.0->httpie==4.0.0b1) Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB) Collecting h11<1.0.0,>=0.11.0 (from urllib3-future<3,>=2.4.901->niquests<4,>=3.4.0->niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Using cached h11-0.14.0-py3-none-any.whl (58 kB) Collecting h2<5.0.0,>=4.0.0 (from urllib3-future<3,>=2.4.901->niquests<4,>=3.4.0->niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Using cached h2-4.1.0-py3-none-any.whl (57 kB) Collecting qh3<1.0.0,>=0.14.0 (from urllib3-future<3,>=2.4.901->niquests<4,>=3.4.0->niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Using cached qh3-0.14.0-cp37-abi3-macosx_11_0_arm64.whl.metadata (4.8 kB) Collecting python-socks<3.0,>=2.0 (from urllib3-future[socks]<3,>=2.4.901; extra == "socks"->niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Using cached python_socks-2.4.4-py3-none-any.whl.metadata (7.1 kB) Collecting hyperframe<7,>=6.0 (from h2<5.0.0,>=4.0.0->urllib3-future<3,>=2.4.901->niquests<4,>=3.4.0->niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Using cached hyperframe-6.0.1-py3-none-any.whl (12 kB) Collecting hpack<5,>=4.0 (from h2<5.0.0,>=4.0.0->urllib3-future<3,>=2.4.901->niquests<4,>=3.4.0->niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Using cached hpack-4.0.0-py3-none-any.whl (32 kB) Collecting cryptography<42.0.0,>=41.0.0 (from qh3<1.0.0,>=0.14.0->urllib3-future<3,>=2.4.901->niquests<4,>=3.4.0->niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Using cached cryptography-41.0.7-cp37-abi3-macosx_10_12_universal2.whl.metadata (5.2 kB) Collecting cffi>=1.12 (from cryptography<42.0.0,>=41.0.0->qh3<1.0.0,>=0.14.0->urllib3-future<3,>=2.4.901->niquests<4,>=3.4.0->niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Using cached cffi-1.16.0-cp312-cp312-macosx_11_0_arm64.whl.metadata (1.5 kB) Collecting pycparser (from cffi>=1.12->cryptography<42.0.0,>=41.0.0->qh3<1.0.0,>=0.14.0->urllib3-future<3,>=2.4.901->niquests<4,>=3.4.0->niquests[socks]<4,>=3.4.0->httpie==4.0.0b1) Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB) Using cached charset_normalizer-3.3.2-cp312-cp312-macosx_11_0_arm64.whl (119 kB) Downloading niquests-3.4.1-py3-none-any.whl (94 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 94.5/94.5 kB 1.2 MB/s eta 0:00:00 Using cached pygments-2.17.2-py3-none-any.whl (1.2 MB) Using cached rich-13.7.0-py3-none-any.whl (240 kB) Using cached setuptools-69.0.3-py3-none-any.whl (819 kB) Using cached idna-3.6-py3-none-any.whl (61 kB) Using cached kiss_headers-2.4.3-py3-none-any.whl (43 kB) Using cached markdown_it_py-3.0.0-py3-none-any.whl (87 kB) Downloading urllib3_future-2.4.903-py3-none-any.whl (337 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 338.0/338.0 kB 6.1 MB/s eta 0:00:00 Using cached wassima-1.0.3-cp37-abi3-macosx_11_0_arm64.whl (261 kB) Using cached python_socks-2.4.4-py3-none-any.whl (52 kB) Using cached qh3-0.14.0-cp37-abi3-macosx_11_0_arm64.whl (263 kB) Using cached cryptography-41.0.7-cp37-abi3-macosx_10_12_universal2.whl (5.3 MB) Using cached cffi-1.16.0-cp312-cp312-macosx_11_0_arm64.whl (177 kB) Building wheels for collected packages: httpie Building editable for httpie (pyproject.toml) ... done Created wheel for httpie: filename=httpie-4.0.0b1-0.editable-py3-none-any.whl size=7312 sha256=d77ed3eca64b9478e633b272b9e9142a5ec2ba03fab00b6a83a31c456fef829b Stored in directory: /private/var/folders/hc/ss_swh2s7rscgx4402mjxtv00000gn/T/pip-ephem-wheel-cache-nppdx0j0/wheels/60/bd/76/3d063ddffe013d962cd599bbfd12c29d9a91313e761fc7f76e Successfully built httpie Installing collected packages: python-socks, wassima, setuptools, Pygments, pycparser, mdurl, kiss-headers, idna, hyperframe, hpack, h11, defusedxml, charset-normalizer, markdown-it-py, h2, cffi, rich, cryptography, qh3, urllib3-future, niquests, httpie Successfully installed Pygments-2.17.2 cffi-1.16.0 charset-normalizer-3.3.2 cryptography-41.0.7 defusedxml-0.7.1 h11-0.14.0 h2-4.1.0 hpack-4.0.0 httpie-4.0.0b1 hyperframe-6.0.1 idna-3.6 kiss-headers-2.4.3 markdown-it-py-3.0.0 mdurl-0.1.2 niquests-3.4.1 pycparser-2.21 python-socks-2.4.4 qh3-0.14.0 rich-13.7.0 setuptools-69.0.3 urllib3-future-2.4.903 wassima-1.0.3 ~/C/P/httpie 🐍 tempenv-51b5170562de8 🌱 feature-tryout-niquests 14s ❯ pip list Package Version Editable project location ------------------ ------- ------------------------------- cffi 1.16.0 charset-normalizer 3.3.2 cryptography 41.0.7 defusedxml 0.7.1 h11 0.14.0 h2 4.1.0 hpack 4.0.0 httpie 4.0.0b1 /Users/dwt/Code/Projekte/httpie hyperframe 6.0.1 idna 3.6 kiss-headers 2.4.3 markdown-it-py 3.0.0 mdurl 0.1.2 niquests 3.4.1 pip 23.3.2 pycparser 2.21 Pygments 2.17.2 python-socks 2.4.4 qh3 0.14.0 rich 13.7.0 setuptools 69.0.3 urllib3-future 2.4.903 wassima 1.0.3 ~/C/P/httpie 🐍 tempenv-51b5170562de8 🌱 feature-tryout-niquests ❯ which https /Users/dwt/.virtualenvs/tempenv-51b5170562de8/bin/https ~/C/P/httpie 🐍 tempenv-51b5170562de8 🌱 feature-tryout-niquests ❯ https --http3 --resolver "doh+google://" pie.dev/get HTTP/2 200 OK Access-Control-Allow-Credentials: true Access-Control-Allow-Origin: * Alt-Svc: h3=":443"; ma=86400 Cf-Cache-Status: DYNAMIC Cf-Ray: 842e2f7a4e881c82-AMS Content-Encoding: gzip Content-Type: application/json Date: Tue, 09 Jan 2024 16:51:31 GMT Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=yU2NQKMxVXTqBiwWJaDtlkEUH3k5p4Z6g7%2BhFgmAFrgFXzR1KMqb98mx7Mu5x4LHu3NPz8Wm7NxYanPM4o18syEXQWm9ug2%2BOo6pOjFaMt%2BHmvRAunSgO%2FND"}],"group":"cf-nel","max_age":604800} Server: cloudflare { "args": {}, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip", "Cdn-Loop": "cloudflare", "Cf-Connecting-Ip": "91.65.247.116", "Cf-Ipcountry": "DE", "Cf-Ray": "842e2f7a4e881c82-FRA", "Cf-Visitor": "{\"scheme\":\"https\"}", "Connection": "Keep-Alive", "Host": "pie.dev", "User-Agent": "HTTPie/4.0.0.b1" }, "origin": "91.65.247.116", "url": "https://pie.dev/get" } ~/C/P/httpie 🐍 tempenv-51b5170562de8 🌱 feature-tryout-niquests 3s ❯ https --http3 pie.dev/get HTTP/2 200 OK Access-Control-Allow-Credentials: true Access-Control-Allow-Origin: * Alt-Svc: h3=":443"; ma=86400 Cf-Cache-Status: DYNAMIC Cf-Ray: 842e2feaadf365df-FRA Content-Encoding: gzip Content-Type: application/json Date: Tue, 09 Jan 2024 16:51:49 GMT Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=oymUHuIvuaPYe9IGgqJnfHIrH6FGoLgcwv%2Bt3Mrkv%2F0%2F4ejdbfgXqUEDsCRESYYv6MSRxbvr84jZnsPyVc5K12ZiaZB%2FCssOdpPV08UNm%2BwNqpAlr7Z1PXL7"}],"group":"cf-nel","max_age":604800} Server: cloudflare { "args": {}, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip", "Cdn-Loop": "cloudflare", "Cf-Connecting-Ip": "91.65.247.116", "Cf-Ipcountry": "DE", "Cf-Ray": "842e2feaadf365df-FRA", "Cf-Visitor": "{\"scheme\":\"https\"}", "Connection": "Keep-Alive", "Host": "pie.dev", "User-Agent": "HTTPie/4.0.0.b1" }, "origin": "91.65.247.116", "url": "https://pie.dev/get" } ~/C/P/httpie 🐍 tempenv-51b5170562de8 🌱 feature-tryout-niquests ❯ ls ~/.config/httpie/ version_info.json ```

Ousret commented 8 months ago

OK. I could collect some data using https://github.com/Ousret/httpie-test/actions/runs/7470350480/job/20328900920 I am going to be able to propose a fix for this, hopefully soon. It has to do with the args parser, something seems off.

dwt commented 8 months ago

Oh great, it is reproducible! Please ping me if I can help.

Also: sorry if it might take me a few days to respond.

Ousret commented 8 months ago

Wasn't easy, but ultimately found why. It depended on the system openssl default cipher list, that excluded by accident QUIC ciphers, therefore disabled http/3... It is fixed. You may try again.

dwt commented 8 months ago

yay!

❯ https --http3 pie.dev/get
HTTP/3 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Alt-Svc: h3=":443"; ma=86400
Cf-Cache-Status: DYNAMIC
Cf-Ray: 84678dcdbdec0bdc-AMS
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 16 Jan 2024 15:57:23 GMT
Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=g7XjPz9ElzpYpSXcIk4CyBHzEoVn4VOu80R65aLy6UJ9bLs3H9zT%2FK%2BNiuYRKsqn%2BWQgG6VHnUtChRB9JzUL39l9Tk11yplURAJR3pqdteMkkiytQEEvPDR6"}],"group":"cf-nel","max_age":604800}
Server: cloudflare

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "91.65.247.116",
        "Cf-Ipcountry": "DE",
        "Cf-Ray": "84678dcdbdec0bdc-FRA",
        "Cf-Visitor": "{\"scheme\":\"https\"}",
        "Connection": "Keep-Alive",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/4.0.0.b1"
    },
    "origin": "91.65.247.116",
    "url": "https://pie.dev/get"
}

Indeed this fixes it for me. :)

Ousret commented 8 months ago

Perfect. You can test it more and report back your experience with it at the HTTPie thread if you may.