Add support for httpx as backend

[jakkdl]

First step of #749 as described in https://github.com/aio-libs/aiobotocore/issues/749#issuecomment-1698255533

I was tasked with implementing this, but it's been a bit of a struggle not being very familiar with aiohttp, httpx or aiobotocore - and there being ~zero in-line types. But I think I've fixed enough of the major problems that it's probably useful to share my progress.

There's a bunch of random types added. I can split those off into a separate PR or remove if requested. Likewise for from __future__ import annotations.

TODO:

• exceptions
  • [x] retryable_exceptions: mostly just need to go through all httpx exceptions and decide which ones are fine
  • [ ] I have done a first pass, but it's quite possible there's more of them as I'm unfamiliar with this, so would be great if somebody else took a look through https://pydoc.dev/httpx/latest/classIndex.html
  • [ ] The mapping between httpx exceptions and aiobotocore exceptions can likely be improved. https://github.com/aio-libs/aiobotocore/blob/b19bc091cdca6eb019de9168b90f7c140b711102/aiobotocore/httpsession.py#L478-L534
• [x] proxy support
  • postponed to later PR
  • this was previously handled per-request, but AFAICT you can only configure proxies per-client in httpx. So need to move the logic for it, and cannot use botocore.httpsession.ProxyConfiguration.proxy_[url,headers]_for(request.url)
  • see https://www.encode.io/httpcore/proxies/ for how to handle it per-request
  • raising of ProxyConnectionError is very ugly atm, and probably not "correct"?
  • BOTO_EXPERIMENTAL__ADD_PROXY_HOST_HEADER
  • seems not possible to do when configuring proxies per-client?
• [x] wrap io.IOBase data in a non-sync-iterable async iterable
  • converted to bytes for now.
• [ ] I have added change info to CHANGES.rst

No longer TODOs after changing the scope to implement httpx alongside aiohttp:

• test_patches previously cared about aiohttp. That can probably be retired?
• replace aiohttp with httpx in tests.mock_server.AIOServer?
• The following connector_args now raise NotImplementedError:
  • use_dns_cache: did not find any mentions of dns caches on a quick skim of httpx docs
  • force_close: same. Can maybe find out more by digging into docs on what this option does in aiohttp.
  • resolver: this is an aiohttp.abc.AbstractResolver which is obviously a no-go.
  • raise error for code passing this
  • figure out equivalent functionality for httpx
• url's were previously wrapped with yarl.URL(url, encoding=True). httpx does not support yarl. I don't know what this achieved (maybe the non-normalization??), so skipping it for now.

Some extra tests would probably also be good, but not super critical when we're just implementing httpx alongside aiohttp.

[jakkdl]

I started wondering whether response.StreamingBody should wrap httpx.Response or one of its iterators (aiter_bytes, aiter_text, aiter_lines or aiter_raw), but am now starting to think that maybe it doesn't make sense to have at all and we should just surface the httpx.Response object to the user and let them handle it as they want.

The way that aiohttp.StreamReader behaves is just different enough that providing a translation layer that handles httpx.Response streams the same way becomes quite clunky/inefficient/tricky/very different. StreamingBody.iter_chunks should be done by specifying chunk size when calling httpx.Response.aiter_bytes, and StreamingBody.iter_lines should use httpx.Response.aiter_lines, but the current API does nothing to stop you from reading one chunk, then one byte, but httpx.Response (very reasonably) only lets you initialize the iterators once. Implementing iter_chunks/iter_lines/etc as reading one byte at a time with await anext() on an aiter_raw sounds awful, since there's no read() method that can return a set number of bytes. That in general makes StreamingBody.read() quite clunky to implement.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 80.06231% with 64 lines in your changes are missing coverage. Please review.

Project coverage is 86.13%. Comparing base (a904bd1) to head (af293b0).

Files	Patch %	Lines
tests/test_basic_s3.py	67.53%	25 Missing :warning:
aiobotocore/httpsession.py	79.13%	24 Missing :warning:
tests/conftest.py	82.00%	9 Missing :warning:
aiobotocore/awsrequest.py	68.75%	5 Missing :warning:
tests/python38/test_eventstreams.py	50.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #1085 +/- ## ========================================== - Coverage 86.34% 86.13% -0.22% ========================================== Files 62 62 Lines 5910 6173 +263 ========================================== + Hits 5103 5317 +214 - Misses 807 856 +49 ``` | [Flag](https://app.codecov.io/gh/aio-libs/aiobotocore/pull/1085/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aio-libs) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/aio-libs/aiobotocore/pull/1085/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aio-libs) | `86.13% <80.06%> (-0.22%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aio-libs#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[jakkdl]

Whooooo, all tests are passing!!!! though I did an ugly with test_non_normalized_key_paths - I understand nothing about that test so I currently made the test pass if httpx returns a normalized path.

current TODOs:

• [x] I should add a command-line parameter that sets the http backend to be tested, so I can set up a CI environment without httpx installed to make sure that works.
• [x] Retryable exceptions.
  • [ ] Maybe try to write a test for it
• [x] figure out the branches in convert_to_response_dict.
  • I think they're fine?
• [x] ~~figure out proxies, or ~~raise NotImplementedError.
  • [x] There is at least one test that sorta checks it so if raising I need to work around it.
• [x] Maybe add test for http_session_cls
• [ ] Add documentation - RTD is broken?

codecov is very sad, but most of that is due to me duplicating code that wasn't covered to start with, or extending tests that aren't run in CI. I'll try to make it very slightly less sad, but making it completely unsad is very much out of scope for this PR.

Likewise RTD is failing ... and I think that's unrelated to the PR?

[jakkdl]

@thejcannon @aneeshusa if you wanna do a review pass

[jakkdl]

Hey @thehesiod what's the feeling on this? It is turning out to be a messier and more disruptive change than initially thought in #749. I can pull out some of the changes to a separate PR to make this a bit smaller at least

[thehesiod]

hey sorry been down with a cold, will look asap. I don't mind big PRs

[thehesiod]

btw check out: https://awslabs.github.io/aws-crt-python/api/http.html#awscrt.http.HttpClientConnection perhaps we should go in that direction so it will be complete AWS impl

[thehesiod]

discussion here: https://github.com/aio-libs/aiobotocore/discussions/1106

[thehesiod]

back from my trip, will look asap, also need to add Jacob as a helper to review these, so much to do, so little time ;)