Consider Requests' Inclusion in Python 3.5's Standard Library

[kennethreitz]

There's a lot to this, but I'll keep it simple...

Would the Python community, as a whole, benefit from Requests being added into the standard library?

Would love to hear your thoughts and opinions on the subject!

[gamesbrainiac]

Yes, because it simplifies the entire process and does not sacrifice the performance. So yes.

[idan]

• What kind of impact would this have on Requests' ability to evolve and grow?
• Does Requests' release frequency jive with that of Python's? A big difference here might be a good indication that stdlib isn't the right home for Requests.

[Lukasa]

Let's CC some people whose opinions we care strongly about:

@shazow @kevinburke @dstufft @alex @sigmavirus24

[swizzard]

What happened to "the standard lib is where things go to die"?

[offbyone]

The release cadence question is a good one; I'd be concerned about losing request's ability to evolve freely. That said, perhaps a requests core library would be a good fit.

[Lukasa]

My 2 $CURRENCY's worth:

I would be disinclined to do it. I think being outside the standard library has given us the freedom to make choices that benefit our users without being stuck behind core dev's policies for version support and release. It allows us to respectfully disagree with the priorities of core dev. And it allows us to make decisions that are ideological, which has been the lifeblood of this project for more than three years.

I think the reality is that if this module enters the standard library the current core team will move on from it. I certainly have little interest following it into the quagmire that is core dev. The most likely to steward requests in the stdlib is @sigmavirus24, and he's just one man. That loss of direction will inevitably lead to an erosion of the library's interface over time, and I think that would be a tragic thing.

The only thing that being in the standard library gives us is our time back. That's a good reason to put it there, if that's what you think it needs, but I don't think we should pretend that it will make the library or the Python ecosystem any better.

[dstufft]

I don't think you actually can add requests to the stdlib without first adding chardet and urllib3 or removing your dependency on them. There's also the thing where Python doesn't want to ship a CA Bundle so you'd need to modify it so that it uses the system platform bundle as Python does naturally now.

Besides that, I'm not sure. It would certainly make it easier to get requests, however part of my work on pip is basically to make it so it's easy to get things like requests without needing to add it to the stdlib. On top of that it's also somewhat confusing to have multiple ways to make HTTP requests in the Python stdlib. The unification of urllib and urllib2 was a good thing in the Python stdlib and I don't think that re-adding that with urllib.request and "requests" is a good idea either. Finally I don't think it would actually help many people, this would only go into 3.5+ so anyone who wants to use requests would have to either use the version that is on PyPI or making 3.5 their minimum supported Python version which I don't think is a realistic thing to happen in any near future.

[tylerdave]

While I think having Requests in the standard library would help new users I don't know that it would help the Python community as a whole. Experienced users know to use Requests and know how to install it.

I'd be especially concerned with the chilling effect it would have on new development as others would be disinclined to contribute, knowing that they wouldn't see their contributed changes in an easily-usable release for a long time.

What about the middle ground of making Python distributions ship with it installed by default?

[hynek]

No it wouldn’t.

requests is absolutely unsuitable for stdlib inclusion for the many reasons stated before me. The urllib3 dependency alone is a complete showstopper; we don’t want it to got to die in the stdlib.

What would be useful is to add something basic and similar to requests built on top of stdlib’s current http resources that allows users to do simple get/post requests to https without practicing blood magic.

Also kind reminder that it would be added to Python 3 only. :) (and not earlier than Python 3.6).

Are you getting tired of maintaining it Kenneth? ;)

[alex]

I don't think we can even begin to discuss this question without someone saying what becomes of httplib, urllib, and friends. Adding requests without addressing the complexity of choice is, I think, worse than the answer "ignore teh stdlib, just use requests". It's a regression to the days of urllib, urllib2.

[sigmavirus24]

I think the reality is that if this module enters the standard library the current core team will move on from it. I certainly have little interest following it into the quagmire that is core dev. The most likely to steward requests in the stdlib is @sigmavirus24, and he's just one man. That loss of direction will inevitably lead to an erosion of the library's interface over time, and I think that would be a tragic thing.

I would wander into the stdlib to try to help, but given the fact that exactly one of I don't know how many previous patchsets I've submitted has been accepted and one other reviewed makes me wary of wanting to bother with that process. I know the core devs are entirely swamped by more important things. I also know someone else has decided randomly that they want to maintain httplib/http but they're clearly not suited for the job (yet) and I don't have the patience to work on httplib when patches that both @Lukasa and I sit around, unreviewed, and not cared about (when they fix pressing issues with the library).

I'd probably end up just forking requests to continue using it.

requests is absolutely unsuitable for stdlib inclusion for the many reasons stated before me. The urllib3 dependency alone is a complete showstopper; we don’t want it to got to die in the stdlib.

It's always been a contention of @kennethreitz (and therefore, the project as a whole) that urllib3 is an implementation detail. Many of requests' biggest features are handled entirely by urllib3, but it doesn't mean they couldn't be reimplemented with care into truly dependency-less library.

Regarding the chardet dependency: it's been nothing but a headache to us (and to me specifically). It used to have separate codebases for py2 and py3 until I got it into a single codebase library (which has only in the last several months been merged back into chardet proper). The library is slow and a huge memory hog (which angers many people to the point of yelling at us here on the issue tracker). It's not entirely accurate and Mozilla's universalchardet that it is modeled after has all but been abandoned by Mozilla. So removing chardet would probably be a net positive anyway.

Regarding whether we should do this or not, I'm frankly unconcerned. Whatever would be in the stdlib would end up being requests in API only. The Python 3 adoption rate is slow enough that I don't think people will be meaningfully affected by this for the next N years (where N is the globally unknown number of years for 3.5 to be used in production by corporations).

And like I said, I'd probably end up just forking requests or using urllib3 directly at that point.

[kennethreitz]

I discussed this at length with Guido the other day — chardet would have to be included first. I think that urllib3 and requests could be included into the http package together.

However, I'm very inclined to agree with @hynek and @dstufft. Perhaps requests is fine just the way it is :)

[kennethreitz]

Either way, if you have an opinion that you'd like to share, you are welcome to share it here (or anytime really).

:sparkles: :cake: :sparkles:

[alex]

Also, adding a new http module to the stdlib with no asyncio-story seems bonkers to me.

On Sun Jan 25 2015 at 1:15:51 PM Kenneth Reitz notifications@github.com wrote:

Either way, if you have an opinion that you'd like to share, you are welcome to share it here (or anytime really).

[image: :sparkles:] [image: :cake:] [image: :sparkles:]

— Reply to this email directly or view it on GitHub https://github.com/kennethreitz/requests/issues/2424#issuecomment-71384152 .

[tBaxter]

I don't think we can even begin to discuss this question without someone saying what becomes of httplib, urllib, and friends

This is my issue. I think the current confusing state of affairs has to be sorted out first.

[leto]

:+1: :metal:

[sigmavirus24]

Just to be clear, my above comment about re-implementing urllib3 for inclusion in the stdlib shouldn't be taken lightly. The amount of work necessary to do that would be immense because urllib3 is the product of (likely 10 or more) years of developer work.

[shazow]

I too have spoken with Guido about tossing urllib3 into the stdlib some years ago with the conclusion that it's not a great idea, but I'm fairly neutral about it at this point.

urllib3's API has been mostly-stable and pretty much completely backwards compatible for several years now. Its' pace is possibly even slower than that of the stdlib today, with the vast majority of changes being minor fixes or security improvements (with occasional backwards-compatible feature additions like granular timeouts/retries). If somebody really wanted to try and get urllib3 into the standard library, I don't think it's a terrible idea—it's just not the best idea.

(I'm not speaking for requests, as it moves at a very different pace with different goals than urllib3.)

The best idea, in my opinion, would be for the PSF to hire (or maybe Kickstart or something) 1-3 developers to build out a brand new http library on top of asyncio with HTTP/2 support with heavy inspiration from urllib3, requests, and hyper. I'd be happy to see as much code taken verbatim as possible but laid out in a consistent, modular, and reusable manner. Ideally target Python 4 or something, and get rid of all the urllibs and httplibs. I expect this would be 6-9mo of hard work, but possibly more.

The very worst part about urllib3, which I'd love to see replaced if somebody attempts to rewrite it per @sigmavirus24's suggestion, is that it depends on httplib. urllib3's functionality is substantially limited with lots of code spent working around shortcomings of httplib. Though if HTTP/2 support would be taken seriously in this goal, then the scope of re-implementing HTTP/1.1 would be a very comforting fraction of the work required.

Many PyCons ago, a bunch of us met up and whiteboarded a layout of a brand new http library that refactors all the pieces into the ideal arrangement we could imagine at the time. I'd be happy to dig up these notes if anyone is going to attempt this.

[Lukasa]

+1 @shazow

Again, if anyone finds themselves with the time and inclination to take on that fairly large project, I've sketched out a putative API design that might make a good starting point.

[dcramer]

Yes because the only way I will ever allow requests as a dependency is if its in stdlib.

That said, urllib3 contains the features that people really want, so having that in stdlib is enough for me

[dstufft]

Do you not use any dependencies?

[dcramer]

@dstufft this is in projects that generally don't, where everyone can't be bothered to figure out how to use urllib. (people aren't adding it as a dep because of ssl/etc generally, but out of laziness)

[dcramer]

@dstufft also multi-version deps basically make it hard to use things in libraries. You probably want to use requests in your project and if we require it then there's a potential for a world of hurt if API changes happen in versions.

[sigmavirus24]

While I appreciate some people wanting to develop dependencies without dependencies on other software that hasn't changed its API in years, this isn't the place for the discussion.

[dcramer]

@sigmavirus24 I disagree. requests has had its API change in the past. APIs change, thats why we have versioning, thats why dependencies are complex. This is a perfect case for that discussion because requests is a dependency in a lot of projects.

If you move into the stdlib the API must be stable.

[sigmavirus24]

@dcramer the API broke exactly once, in 1.0. APIs do change, but requests' API isn't planning any changes either. The only change we've had is adding the json parameter which doesn't break anything. You can keep trying to accuse us of breaking the API too much, but when projects like OpenStack have had requirements defined as >=1.2.3 for a long time, I think that says a lot about the stability of requests. The API has been stable, precisely because after we cut 1.0 we rejected all new additions to the API (with the obvious recent exception of adding a json param) and we've been very strict about not breaking the API. If you're not a consumer of requests, you wouldn't know this though. So I don't take your ignorance personally.

[sigmavirus24]

Further if the stdlib API is supposedly so stable, explain why argparse broke it's public API between Python 3.3 and 3.4?

[dcramer]

@sigmavirus24 you're now purely turning this into an API debate. I was just pointing out the reason I dont include it is because it can change, and everyone uses it, and everyone uses different versions. It's great that you guys never change your API but I have no desire or time to follow it or assume thats true.

[dstufft]

You know Python changes it's API too, quite often actually, sometimes in very major ways, perhaps you've heard of Python 3?

[dcramer]

Well i'm leaving this debate. I made my opinions clear. Not sure what you're all bitching about.

[ncoghlan]

I think some key questions to answer are:

1. How would you want the standard documentation (including the tutorial) to change? The current standard library HTTP APIs date from an era where abstrating away the details of competing protocols (e.g. FTP vs HTTP) was considered a desirable approach. In the subsequent decade and a half, the web development community has converged on HTTPS+JSON as the preferred approach to request/response style client/server interaction for most use cases other than remote command execution (which uses either SSH or Windows RCP), and the requests API is the de facto standard client implementation of that model for modern Python applications.
2. Do you want the requests API to be upgraded from a de facto standard to a de jure standard, such that it automatically gets included in all CPython redistribution channels, gets backed by CPython's (and our commercial redistributors) long term support guarantees, as well as all "standard library only" educational activities? (The latter are still very common, as the criteria for use in corporate environments often includes support and IP guarantees, which CPython has, but requests doesn't. Under the current situation, many users simply won't consider requests as an option because it's far too much work for them to get it accredited for use)
3. What other modules in the standard library could be improved by having a requests like API to build on?
4. Would it be better to keep requests itself unchanged as a version-independent implementation of the API, and instead add a new requests inspired API to the standard library, akin to the way Guido approached the ayncio standardisation work?

As far as the idea of PSF managed development work goes, I'd be heavily against that, as we don't have the management infrastructure in place to handle that kind of thing. A grant proposal suggesting the PSF contribute to a crowdfunded effort in this area would be a different story (no guarantees without a specific proposal to review, but it's certainly an idea we'd be amenable to discussing).

[ncoghlan]

Note that some vendors may already redistribute requests, but "do they also provide support for requests?" becomes a separate question we have to ask of a vendor or platform provider. Thus in a long term risk management context (think years and decades, rather than weeks or months) depending on it means we either have to assume the risk and responsibility of self-support if upstream gets bored and move on to something else, or else accept a potential reduction in the degree of choice we have in platform providers.

With the standard library, we generally don't have that concern - while redistributors occasionally break things, in a commercial context, vendors breaking stuff that works upstream gives you quite a lot of leverage to get the offending vendor to fix things.

[ncoghlan]

Oh, one other key question to answer before volunteering to actually maintain stuff in the standard library: are you prepared to accept the responsibility of shipping software that helps power half the world's stock exchanges, is one of the most popular languages for corporate infrastructure, one of the most popular languages for scientific programming (including being used for trajectory planning on inter-planetary space missions), one of the most popular languages for web development, and one of the key languages being employed in new computer literacy initiatives in educational institutions?

Are you also prepared to do that while having folks working on non-critical services where very low reliability levels (often less than 99% uptime) are perfectly acceptable complain that your profound trust issues are unwarranted, and merely a matter of stupid hidebound politics that they can't be bothered concerning themselves with?

[ncoghlan]

Also, a mistaken assumption I'd like to correct: the vast majority of Python programmers likely won't even have heard of pip, let alone requests. These are the folks running scripts in the system Python on long term support Linux releases, the folks most open source developers are quick to express unutterable contempt for, without pausing to consider what circumstances might be at play to make their approach a good idea.

The adoption cycle times in this community are inherently measured in years. It's only a vanishingly small fraction of the industry that is currently running sufficiently rigorous CI to be able to go faster, and most folks in that fraction aren't interested in slowing down long enough to listen to the concerns of those with large swathes of existing infrastructure that we need to manage and modernise.

That post-chasm portion of the technology adoption curve are the ones we reach when we say "yes, this approach is now sufficiently mature that we can push it, or something based on it, into the standard library to help make it a truly universal assumption".

[ncoghlan]

As a more concrete example of how the "open source community" filter bubble can skew our perspective: Jenkins holds more than a 70% market share in corporate CI deployments.

Be very, very cautious in relying on intuitions about what "everybody knows", when you're basing that perspective on individuals and organisations that are heavily involved in open source. The vast majority of software development is still custom work to meet the needs of a particular organisation, and the vast majority of folks doing that are still getting their tools and information from commercial vendors rather than the open source community.

[sigmavirus24]

@dcramer Not sure what you're all bitching about.

Very appropriate language for this debate. When legitimate counters are made to your position, you use a slur meant to demean women. Par for the course though. Moving on,

---

@ncoghlan

Re point 1: I think the documentation would be drastically simplified with requests (requests-alike) in the stdlib. One of the first things I do when learning a new language is figure out how do HTTP. Having that featured is something the guide would benefit from regardless.

Re point 2: There's a difference between the API and the library being de facto vs. de jure. The API could easily be provided by the standard library. I think your concern about support would be more aimed towards requests (the code) being included.

Re points 3 & 4: I'm not sure that's something to be discussed here. Maybe python-ideas would be better.

As far as the idea of PSF managed development work goes, I'd be heavily against that, as we don't have the management infrastructure in place to handle that kind of thing.

That's interesting. I didn't think it was a probability but it'd be great to have something better than http(lib).

With the standard library, we generally don't have that concern - while redistributors occasionally break things, in a commercial context, vendors breaking stuff that works upstream gives you quite a lot of leverage to get the offending vendor to fix things.

I'm not sure what leverage you're talking about. I've seen ensurepip, venv, and other things broken by Debian and other redistributors in CPython. That's tangential to this discussion though.

Oh, one other key question to answer before volunteering to actually maintain stuff in the standard library: are you prepared to accept the responsibility of shipping software that helps power half the world's stock exchanges, is one of the most popular languages for corporate infrastructure, one of the most popular languages for scientific programming (including being used for trajectory planning on inter-planetary space missions), one of the most popular languages for web development, and one of the key languages being employed in new computer literacy initiatives in educational institutions?

As already mentioned, most of the people currently involved wouldn't continue with the project. I'd probably be the only person but given my track record with upstream CPython development, it wouldn't be productive leaving that burden to the existing (and other future) core developers.

That post-chasm portion of the technology adoption curve are the ones we reach when we say "yes, this approach is now sufficiently mature that we can push it, or something based on it, into the standard library to help make it a truly universal assumption".

The reality is that those people will never catch up though, no? People are still running software on Python 2.4 and 2.5. F5's load balancers still only support Python 2.5. 2.7 will be in use probably until the end of my natural life (which I hope will be quite long). Are these really the people this decision will affect most strongly? Those same people you describe may never make the leap to Python 3. And currently, it still is a leap. Maybe by the time they've decided to consider it, Python 3.8 or 3.9 or 4.2 will be out and will be much less of a hassle for them.

[ncoghlan]

Right, my main point is that the objective of standard library inclusion is very different from that of the more common task of providing an open source library for use by other open source developers. If the requests team choose to continue providing only the independently maintained library (a community service I am immensely grateful for, you folks do great work!), and don't back a push for standardisation of the API, then we'll be relying on redistributors like RHEL/Fedora/CentOS, Debian/Ubuntu, ActiveState, Enthought and Continuum Analytics to pick up the module and standardise on it anyway, such that folks can assume it will always be available (or at least often enough for them to be happy relying on the API in single file scripts that aren't fully packaged with appropriate dependency declarations). However, most introductory documentation will likely still steer people towards doing HTTP the standard library way, so whether folks get taught "HTTP for Python, 2001 edition" or "HTTP for Python, 2015 edition" will depend on whether they pick it up from a "standard library only" source, or one that encompasses recommended third party modules.

As with virtualenv and PEP 405 though, or Twisted+Tornado vs asyncio, or ipaddr vs ipaddress, I don't believe it makes sense to include requests itself in the standard library. Rather, I think it makes more sense to back inclusion of a requests inspired API that leaves out all the cross version compatibility code, the certificate bundling, etc, etc, and simply brings the default APIs and documentation for handling authenticated HTTP requests up to 2015 standards. Then in 2030, we'll be complaining about how archaic the requests API design is by 2030 standards ("HTTP? Who uses that any more?!?!"), but that's OK, it's just how these cycles work (until first AJAX and then JSON came along, XML-RPC was king of the hill). If you get 5 years out of an API design before people start complaining about it being dated you're doing pretty well, 10 is impressive, and 15 is truly spectacular.

The main thing needed to start that process is someone sufficiently motivated by the idea of having the requests API available everywhere to champion the standardisation effort, do the implementation work, and aim to commit to at least the first few years of stdlib maintenance. The alternative is to continue relying heavily on redistributors to reach developers that aren't willing and/or able to download and run arbitrary code from the internet (a category that includes everyone in any industry with strict regulatory due diligence requirements, as well as anyone working for other organisations with a similarly low appetite for risk).

[ncoghlan]

Regarding some of the other points, out current leverage with Debian is relatively weak, while the leverage is better with commercially supported redistribution where paying customers are able to complain directly about things being broken relative to the expectations set by us as an upstream development community.

In terms of update cycles, one of the key things upstream standardisation (even in Python 3) conveys is community consensus. At that, libraries which are "backports" of Python 3 features become much easier to justify bundling with Python 2. I'd personally still like to see a sumo release of Python 2 which does exactly that bundling (unittest2, subprocess32, enum34, contexlib2, trollius, etc), but that's a whole separate political battle in its own right, especially in terms of persuading folks that the interest and resources to maintain such a redistributor independent sumo distribution until 2020 are actually available.

[kennethreitz]

@dcramer thank you for giving us your time — it's really appreciated :heart:

[kennethreitz]

@sigmavirus24 all is well :)

[kevinburke]

I don't have much to add on the stdlib inclusion front, other than, it would be neat to look at some PEPs or threads or talks about what should go in the Python stdlib, and maybe to look back at the last year or so's development from the perspective of, "how would handling this issue be different if this was a module in the standard library".

Since this may become the de facto thread people look at when they say "what should we add/change when considering adding Requests to the stdlib", I'd like to give another shout for updating the exception hierarchy. I usually have two questions when a request fails - a) What are the implications? and b) Can I retry the request safely? A DNS lookup failure and a broken pipe have very different implications, but requests currently groups both as a ConnectionError. I've detailed some of the work involved here: https://gist.github.com/kevinburke/b98e053a4bf9835c67bb

Happy to discuss more with interested folks.

Kevin Burke phone: 925.271.7005 | twentymilliseconds.com

On Sun, Jan 25, 2015 at 8:15 PM, ncoghlan notifications@github.com wrote:

As a more concrete example of how the "open source community" filter bubble can skew our perspective: Jenkins holds more than a 70% market share in corporate CI deployments.

Be very, very cautious in relying on intuitions about what "everybody knows", when you're basing that perspective on individuals and organisations that are heavily involved in open source. The vast majority of software development is still custom work to meet the needs of a particular organisation, and the vast majority of folks doing that are still getting their tools and information from commercial vendors rather than the open source community.

— Reply to this email directly or view it on GitHub https://github.com/kennethreitz/requests/issues/2424#issuecomment-71413074 .

[bronger]

FWIW, I think the wording on http://docs.python-requests.org/en/latest/dev/philosophy/,

Essentially, the standard library is where a library goes to die.
It is appropriate for a module to be included when active 
development is no longer necessary.

may get misinterpretated. To me, it sounds like a pejorative attitude towards the Standard Library, which of course does not consist of dead libraries. I was irritated and refrained from using requests, until I hit this page, which is an interesting discussion and much better than the wording above.