search

encode / httpx

A next generation HTTP client for Python. 🦋
https://www.python-httpx.org/
BSD 3-Clause "New" or "Revised" License
13.24k stars 839 forks source link

Version 1.0 #947

Open tomchristie opened 4 years ago

tomchristie commented 4 years ago

Update: Everything we need to consider ought to now be milestoned... https://github.com/encode/httpx/milestone/1

API Reference

Here's a high-level API reference...

Helper functions

request, stream, get, options, head, post, put, patch, delete

Clients

Client, AsyncClient

Models

Response, Request, URL, QueryParams, Headers, Cookies

Configuration

Limits, Proxy, Timeout

Authentication

Auth, BasicAuth, DigestAuth

Transports

ASGITransport, WSGITransport

Status Codes

codes

Exceptions

Dependencies

Optionals: pip install httpx[http2], pip install httpx[brotli]

florimondmanca commented 4 years ago

I’d like to add a discussion point about our position wrt type hints would be. There’s been a couple of issues asking for them to be public API mainly for the use of third party library authors. I don’t have a strong position on this yet, but wondering if type hints (and more generally third party lib authoring experience, although that’s pretty broad) should be in our scope for 1.0?

tomchristie commented 4 years ago

Good call yup, we need to discuss that for sure!

yeraydiazdiaz commented 4 years ago

We have a version 1.0 milestone. Should we formalize these points into issues and add them?

tomchristie commented 4 years ago

The proxy DEFAULT_MODE discussion needs to be part of this too.

tomchristie commented 4 years ago

@yeraydiazdiaz Sure thing, yup.

pikeas commented 4 years ago

Having recently used httpx, I was very surprised that there's no rate-limiting (requests per second, not just max connections) built in. This greatly reduces the utility of the async mode.

tomchristie commented 4 years ago

Okay. Sure we can have a discussion about that, tho it's not strictly a 1.0 blocker.

florimondmanca commented 4 years ago

Something else to potentially consider for 1.0 that came to my mind: #302, aka dropping certifi in favor of the system defaults. I have a different approach in mind that what was originally suggested in the issue, will comment there to see if it makes sense.

bwright1558 commented 4 years ago

I've had my eyes on this repository for a while and it looks very promising. The only thing preventing me from using this library is the lack of support for socks5 proxies. Gotta have this for connecting to Tor. It's also a lot easier to spin up a socks proxy server than it is an http proxy server (in my opinion). There's already an open issue: #203. There's also a PR in httpcore: encode/httpcore#51. Would be nice to see this in a 1.0 release.

florimondmanca commented 4 years ago

@bwright1558 From my POV, SOCKS proxy support is a nice-to-have, but technically not a blocker for 1.0 (we can probably add it later as an incremental new feature, without having to break any existing API).

Note that there's been loads of work by Yeray and @sethmlarson to pull off a sans-io implementation, and as you mentioned Yeray started the integration within httpcore (which is indeed almost only additive). If you'd like to help out I'm sure Yeray will appreciate early-stage user feedback there, e.g. see if the API makes sense, etc. :-)

yeraydiazdiaz commented 4 years ago

I agree with @florimondmanca in that SOCKS support is not a blocker for 1.0, though I certainly see it coming sooner rather than later 🤞

Personally I think the two most requested features for 1.0 are:

To me the latter feels more of a nice-to-have for 1.0, it could even be argued that it's better to leave it as a separate library completely.

florimondmanca commented 4 years ago

I think those are two interesting areas, but are they blockers for 1.0?

I’m still not super convinced we need to have retries built in (libraries like tenacity solve this problem brilliantly). But if we do want some basic functionality then I’d definitely make the case for making retries opt in (which makes it easier to bring in higher control via eg tenacity), in which case this would be a non breaking addition in all cases, so not a 1.0 blocker.

As for throttling, it seems we currently have expectations set wrong wrt pool limits. A lot of people seem to expect those to behave like throttling, when it’s really more about resource limiting. I guess we could address this with some docs revamp about pool limits (I remember Tom drafted a PR for this), and then perhaps recommend focused solutions like aiometer more prominently - at least as a first approach. (FWIW aiometer already has an HTTPX usage example in its README.) Like retries, I don’t see why we wouldn’t be able to bring this in incrementally (throttling should be opt in too), so not a 1.0 blocker in my mind.

StephenBrown2 commented 4 years ago

Probably not a strong argument, but retries are "built in" to Requests due to it relying on urllib3 for that, so adding it would allow a smoother transition from requests.

But, I will also concede it's not a requirement for meeting the HTTP client spec (aside from the Retry-After header being a thing...)

hugovk commented 4 years ago

Not directly an httpx issue, but a heads-up that the httpx mock library RESPX doesn't yet support httpx 0.13:

P0oOOOo0YA commented 4 years ago

So you're going to release 1.0 without socks v4/5 support? :-1:

tomchristie commented 4 years ago

So you're going to release 1.0 without socks v4/5 support? 👎

Not acceptable. You need to rethink how you approach folks in the community - that’s a massively entitled and disrespectful way to approach your first interaction with a project.

Here’s a better approach:

Q: “Hi folks! Is socks v4/5 a priority? Does that come before or after 1.0?”

A: “1.0 is about API stability. Socks support might come either before or after that, but it’s not a blocker for us indicating that the project is API stable.”

P0oOOOo0YA commented 4 years ago

A: “1.0 is about API stability.

Hi folks! Sorry for not so well first impact but I think 1.0 is about minimum functionality as well. Maybe you're right and i'm a perfectionist to consider supporting proxies (both HTTP(s) & socks v4/5) the minimum requirement of an HTTP client. Anyway congratulations

hugovk commented 4 years ago

Not directly an httpx issue, but a heads-up that the httpx mock library RESPX doesn't yet support httpx 0.13:

The newly-released RESPX 0.11 now supports HTTPX 0.13 🎉

pawamoy commented 4 years ago

Hello everyone, how is the 1.0 going 🙂 ? I would like to use HTTPX at work but my client requires software to be in a somewhat "stable" status, and the first note of the first page of the docs indicates that HTTPX should still be considered beta, so right now they won't have it.

Is there a way I could help you releasing version 1.0? Maybe start working on the issues in the 1.0 milestone? I'd appreciate some guidance as I didn't contribute to this project yet, and don't want to be a hindrance to the development 🙂 I'm also not particularly knowledgeable in HTTP protocol matters 😅 Let me know if there's anything I can help with 🙂

florimondmanca commented 4 years ago

Hi @pawamoy,

I understand that the currently advertised beta status could be a hindrance to spreading HTTPX in environments that require stability (I personally would like to start replacing Requests with HTTPX at work 😉), but for now that's how things are. HTTPX will will promoted to "stable" with 1.0.

I hope you'll understand that I'm personally sort of pushing back on asks like this since quality of design requires lots of careful thinking, and that means time. We shouldn't rush things.

So I personally can't even say when 1.0 will be. "Somewhere mid/late 2020" would be my fairest guess for now. We should be releasing 0.14 soon, and that's planned to be our last minor before 1.0.

If you'd like to help, yup, feel free to look over items marked for the 1.0 milestone, since those are required before reaching 1.0. In terms of guidance well, give yourself some time to navigate and understand the code at a high level, then dig deeper as you need. :-) Our contributing guide should help you get up and running as far as tooling and development workflow is concerned.

Cheers!

pawamoy commented 4 years ago

@florimondmanca of course, I was not asking for a 1.0 release just for the sake of it, but rather if I could help to implement the things that are required for the 1.0 release. I totally understand the need for careful thinking, and I don't want you to rush things! Thanks for your answer, I'll then see what I can do :slightly_smiling_face:

aalvrz commented 3 years ago

I do not have any requirement to use httpx at work, but I have been watching this project for some time and I must say I am really amazed by the amount of work, discussion and thought put into everything in it :heart: . I am sure the 1.0 release will be glorious :smiley: :rocket:

nkitsaini commented 3 years ago

Should we update the README.md to reflect new estimate for 1.0 release ?

florimondmanca commented 3 years ago

@nkitsaini "Some time in 2021"? I guess we could aim for that. 😄 Feel free to send a PR if you're up for it!

manatlan commented 3 years ago

I vote for SOCKS proxy support too! (aiohttp & requests support them ... if it's not here : I doubt that it can be used/useful in the pro world (others programs/tools will not switch to httpx without that (it's my case)) I say that, I say nothing ... I will continue to use aiohttp til httpx supports it ;-(

emil45 commented 3 years ago

I vote for SOCKS proxy support too! (aiohttp & requests support them ... if it's not here : I doubt that it can be used/useful in the pro world (others programs/tools will not switch to httpx without that (it's my case)) I say that, I say nothing ... I will continue to use aiohttp til httpx supports it ;-(

+1

tomchristie commented 3 years ago

I'm very keen on adding SOCK proxy support, and I have been looking at it as part of the revised httpcore work I'm doing. However, there's no need for us that work to block declaring our library as API stable, with a 1.0 release.

I've just raised two discussions, which I'm considering the final blockers for a 1.0 release...

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

tomchristie commented 2 years ago

Thanks @stale.

Brief synopsis here. I think we're actually basically API stable for 1.0. I've got a bit of work on dropping a couple of our package requirements.

For example see https://github.com/encode/httpx/discussions/2083 to drop charset-normalizer.

Plus we ought think about any particular docs we want to update prior to a 1.0. (Eg outlining our deprecation policy. Also see #1797)

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

michaeloliverx commented 2 years ago

The official Azure SDK for Python is interested in adding a httpx based transport see https://github.com/Azure/azure-sdk-for-python/issues/11279 but have stated they won't proceed until httpx cuts a stable 1.0 release.

kashifkhan commented 1 year ago

I just wanted to provide an update - I work on the Azure SDK for Python team and we released the azure core experimental library where experimental features that aren't ready yet for stable are made available. The most recent release has added support for httpx as a transport, so folks can use it when using the azure libraries. There is a sample here that shows how to use with text analytics .

Please give it a go and we would appreciate any feedback yall have 🙂

bbkane commented 1 year ago

@kashifkhan , does https://github.com/kashifkhan/azure-sdk-for-python/blob/main/sdk/core/azure-core-experimental/samples/httpx_integration/httpx_transport_sample_async.py use httpx? The transport is named AsyncHttpTransport (not AsyncHttpXTransport) while the sync example transport is named HttpXTransport

kashifkhan commented 1 year ago

@bbkane mistake on my part, thank you for pointing it out, Ill fix it in the morning. It should say AsyncHttpXTransport instead :)

astrojuanlu commented 1 year ago

0.24.0 incoming https://github.com/encode/httpx/pull/2652 maybe 1.0 is next, maybe not :)

urbanogt commented 1 year ago

At Binance we would like to use httpx for our python connectors (i.e. https://github.com/binance/binance-connector-python) but also require a stable 1.0 version.

Happy to help! Hopefully we see that stable version soonish

tomchristie commented 1 year ago

@urbanogt - Thanks, yes I'd also like to see a 1.0 release.

Happy to help!

Funding or code collaboration are both very much welcomed. If you'd like to help with the later, keep your eyes open on the repo and jump in when you can. 😎

tomchristie commented 1 year ago

If pointers are helpful... taking a look at https://github.com/encode/httpx/issues/2788 would be a good one to get resolved.

tomchristie commented 1 year ago

Okay, I'm going to just sketch out the bits that I've had in the back of my mind around this...

Bugs that I might currently consider to be 1.0 blockers.

Actually only one of those is API changing, so they don't necessarily need to be blockers. Can just be regular minor point bugfix releases. But perhaps they're significant enough to be treated as 1.0 blockers anyways?

Documentation refresh.

Not technically a 1.0 blocker in any sense, but if we end up with lots of exposure from the release it's going to be the first thing newcomers see so? We've clearly got scope for a really nice docs overhaul.

Minimising project dependancies.

Our default dependancies are...

We could potentially have a project with a really light default dependancy footprint...

Personally I find this a really attractive option, and can talk it through in detail if needed.

API finessing.

There are a couple of areas where our API could fit together more neatly with some changes.

We clear don't want to add code churn for our userbase. Perhaps we leave these kinds of changes for a future 2.0 version which also has a really nice gentle deprecation path.

Probably none of those are absolutes, and we're in a position for our next releases to be a 1.0 of httpcore and a 1.0 of httpx if we chose to do so.

Thoughts, @encode/maintainers?

Kludex commented 1 year ago

Can we define a versioning policy for the organization, and specify what versions are we going to give support?

Also, given that the previous "1.0 path" didn't go as planed, do you think it would be helpful to be specific on what are the steps to achieve 1.0, in a more objective way?

Since I'm not that involved in HTTPX, I don't have opinions about the feature set.

Not technically a 1.0 blocker in any sense, but if we end up with lots of exposure from the release it's going to be the first thing newcomers see so? We've clearly got scope for a really nice docs overhaul.

Completely unrelated to what you said here, but maybe we can add the sponsors in the homepage on the documentation?

tomchristie commented 1 year ago

but maybe we can add the sponsors in the homepage on the documentation?

That's a smart & valid 1.0 request.

ahopkins commented 1 year ago

We could potentially have a project with a really light default dependancy footprint...

Minimizing dependencies is of course always a win. Especially when you consider some that are not strictly necessary (looking at you anyio). Of course, it does make implementation more complicated and there is the maintenance tradeoff. No strong opinions on this, but I would certainly be interested to hear more about what you had in mind to achieve this.

karpetrosyan commented 1 year ago

Mounts and ssl_context makes perfect sense.

  • The proxies=... API is somewhat overloaded. There are neater approaches that we could take if we switched to a single proxy=... API, and use the mounts={...} for more complex cases.
  • The SSL configuration using verify/cert/trust_env is somewhat overloaded. I can see that a neater approach would be a single ssl_context=... parameter.
  • There's a couple of very minor bits of the URL API that could do with tweaking. .raw_path would probably be better named as .target. Also the .raw property is noted as "to be deprecated".

I believe that a documentation refresh is critical for the 1.0.0 release. Motivated newcomers should see beautiful documentation.

Also, I think we should have 0.26.0 before 1.0.0, so that we can test the changes before releasing 1.0.0.

astrojuanlu commented 1 year ago

I'm just a bystander here but I want to spot a chicken-and-egg problem:

Many shops have stated that they're waiting on httpx 1.0 to adopt it. With adoption could (potentially) come more funding and contributors. Without such funding and contributors, getting to 1.0 might get more difficult.

So, if you want to help httpx in the short term, you probably want to help with some of the bugs @tomchristie has stated, or otherwise refrain from making the scope of the 1.0 release bigger.

samuelcolvin commented 1 year ago

Pydantic would be happy to sponsor, we love using httpx (as well as all the other things we've all learnt from @tomchristie).

I'll set up sponsorship tomorrow.

tomchristie commented 1 year ago

That's immensely generous of you, thanks @samuelcolvin ❤️

tomchristie commented 1 year ago

Can we define a versioning policy

We should make it clear that we follow SEMVER.

We should also recommend that users pin their requirements to at least a major version range.

Informally we can follow a stronger-than-SEMVER policy. If we introduce an incompatible API change in a major version, then we can first have a major version bump that issues a warning with the old usage, but does not break. We'll only then remove the old usage after a second major version bump.

Deprecating formal support of older Python versions should require a version bump.

My concise "where are we at" is... Let's release a 1.0 version with a neat SEMVER policy immenently.

Following up on https://github.com/encode/httpx/issues/947#issuecomment-1715532202 more specifically...

I think once we've got that done we can move to a time-based release schedule, improve our documentation, focus on how we deal with sponsorships, and iteratively improve on the project at a pace that isn't rushed.

Kludex commented 1 year ago

Deprecating formal support of older Python versions should require a version bump.

"version bump" here is not clear if you mean minor or major. By context, it looks like you mean a major?

Informally we can follow a stronger-than-SEMVER policy. If we introduce an incompatible API change in a major version, then we can first have a major version bump that issues a warning with the old usage, but does not break. We'll only then remove the old usage after a second major version bump.

It would be nice if you check https://github.com/encode/starlette/pull/1888/files#diff-8643a6bd412b94d3754ee992dec7139d2db1d1e49902e02584f563f12490f71b , so we can align those thoughts between the encode projects. 🙏

I don't think we should block a 1.0 release on a major refresh. There's a significant amount of work here, so I'd suggest we'd be better off cutting our 1.0 release and making our versioning policy clear, and then following up with a later docs refresh. A few minor changes would be sufficient...

It shouldn't be a blocker, clearly. But... Are we in a hurry?

tomchristie commented 1 year ago

"version bump" here is not clear if you mean minor or major. By context, it looks like you mean a major?

That was my intent here, yup. I'm agnostic on it tho. Maybe a simple policy for both starlette and httpx would be "Older Python versions will always continue to be supported at least until their EOL date. Python versions will only be dropped in either a new major or minor release, and will not be dropped in patch releases." (?)

But... Are we in a hurry?

No. It's just a trade-off we're making if we're pushing back on the timing. A suggestion would be that we time-box this. For instance, from here we could very reasonably commit to releasing 1.0 no later than the end of October.

Kludex commented 1 year ago

I'm afraid the docs never reaching the state mentioned here if we release 1.0 without them.