search

request / request

🏊🏾 Simplified HTTP request client.
Apache License 2.0
25.67k stars 3.14k forks source link

Request’s Past, Present and Future #3142

Open mikeal opened 5 years ago

mikeal commented 5 years ago

Before I go into the details and reasoning I’ll get straight to the point. The most valuable thing request can do for the JavaScript ecosystem is to go into maintenance mode and stop considering new features or major releases.

Apologies in advance to the other committers on request that have been doing their best to improve it, but it’s for the best.

2009

The first version of request was one of the first modules ever created for the Node.js ecosystem. The earliest versions were written to APIs that pre-date the standard callback interface, streams, node_modules and npm. For the first few years, request and Node.js evolved together, each learning from the other. As Node.js improved and migrated core interfaces so did request. As request adopted changes to the core http library and streams it also informed improvements like the pipe event (which enabled request’s one line proxy) and one of Core http’s many re-writes (the one I had to write).

npm

request was one of the first modules added to the npm registry. As npm grew so did dependence on request. Even now, when npm is used far more for front-end than back-end work, request remains one of the most depended on modules in the registry. As I write this, 41K modules depend on request and it is downloaded 14 million times a week.

The place request has in the Node.js ecosystem is no longer one of an innovator but of an incumbent. If you Google for how to do something with HTTP in Node.js the examples are likely to show request as the client and express as the server. This has two notably bad effects.

It’s much harder for new libraries accomplishing similar tasks to gain adoption because of the incumbent position request holds over the ecosystem. It’s also very hard to change request in any meaningful way as the change not only may not be adopted by the majority of its dependents but it would put it out of alignment with the thousands of blog posts and stack overflow responses that use request.

Modern JavaScript

The last few years have been dramatic ones in JavaScript. Features people had talked about for years went from ideas, to standards, to features you can reliably depend on in most environments. The speed at which these have been adopted is staggering, mostly thanks to auto-updating browsers and an aggressive Node.js release schedule.

The patterns at the core of request are out of date. A few people might argue with that assessment, and I know who they are so I won’t be surprised, but it’s true. I have often been skeptical of the impact some of these features would have only to find myself adopting them wholesale not long after they are available in only the latest release of Node.js.

There’s a transition happening now in the ecosystem to these patterns. How messy that will be is still up in the air and I’m not going to try and read the tea leafs and figure out what the future looks like in that regard. The question for request is “Do we try to survive through that transition?” A year ago, I thought the answer was obvious and that we would, but now I’m convinced of the opposite.

A version of request written to truly embrace these new language patterns is, effectively, a new module. I’ve explored this space a bit already and have a project I’m quite happy with but it is incompatible with request in every conceivable way. What’s the value in a version of request that is incompatible with the old patterns yet not fully embracing the new ones? What’s the point in being partially compatible when there’s a whole world of new modules, written by new developers, that are re-thinking these problems with these patterns in mind?

The best thing for these new modules is for request to slowly fade away, eventually becoming just another memory of that legacy stack. Taking the position request has now and leveraging it for a bigger share of the next generation of developers would be a disservice to those developers as it would drive them away from better modules that don’t have the burden of request’s history.

Maintenance Mode

Here’s the plan.

reconbot commented 5 years ago

I fully support this, I think a warning message and/or deprecating new releases is in order.

As for the change in process and guidelines, it makes my job a lot easier 👌

simov commented 5 years ago

Very well said @mikeal. I'm pinning this issue to gain more visibility.

reconbot commented 5 years ago

Things we might do - please discuss and volunteer!

analog-nico commented 5 years ago

It makes a lot of sense! I will slowly adopt this policy for the request-promise family as well. Cheers to your important contributions to the node ecosystem!

bajtos commented 5 years ago

deprecate the latest npm package and auto deprecate them on publish

Please be careful about deprecation. As Mikael wrote above, there are 41K modules depending on request. Many of these modules are useful in the current state and work well for their users, but their maintainers may not have time to rework those modules to use something else than request. By deprecating request at install time, you will basically deprecate a large portion of npm module ecosystem.

As I see it, maintenance mode is not the same thing as deprecation.

reconbot commented 5 years ago

I hear you. The full text

deprecate the latest npm package and auto deprecate them on publish via ci (maybe after support has been stopped?)

I think we should eventually deprecate request because I don't want new projects to use it. I've been trying to triage the issues and prs down to a list we can solve, but there are bugs that we cannot fix without a breaking change. Therefore they wont get fixed and new users will have issues.

For example following redirects looses request bodies and cookies, and url parsing to remove relative paths are both bugs but I'm not sure they'll ever be fixed.

Maybe deprecation isn't the right answer, but I don't know how else to approach that.

Does that make sense?

mikeal commented 5 years ago

Let’s just bump the major version when we deprecate. That way most people depending on the project won’t see this error until they try to upgrade to a new major, which means they are actively developing it and really should look for an alternative.

nylen commented 5 years ago

I'm proud to have been part of the history of request. I will also check out bent, it looks interesting, and small, which is more important to me these days.

We’re going to have to remove inactive collaborators and enforce 2fa, because commit rights will effectively become npm publish rights.

Fine to remove me.

JonathanRowell commented 5 years ago

I think we should eventually deprecate request because I don't want new projects to use it.

As a programmer who is VERY grateful for the module and who uses it all the time I WANT to use it on new projects.

CliffS commented 5 years ago

This decision must have been very hard to take but is commendable in the extreme. Well done.

rimiti commented 5 years ago

I'm proud having used this amazing tool. It forced the community to improve. 🙏 If you need help to maintain it do not hesitate to contact me.

pdxmholmes commented 5 years ago

While I respect your decision, I would ask you to consider how much real world, production, code relies on request currently. It's far more than even NPM stats can tell you. I fully understand wanting to move on to a new thing and doing something in a new, more interesting way...this is the JavaScript ecosystem after all, have to chase the new thing. But please consider the amount of time and money you'd be costing professional engineering organizations by wholesale deprecating request. If you want to leave it in maintenance mode, that's fine, but understand that plenty of people have absolutely no practical reason to change libraries. Forcing people to change because of ideology is going to lead to frustration.

Regardless, thanks for the hard work everyone has put in to this library.

kibertoad commented 5 years ago

I wonder what library could be considered modern and recommended now. Superagent is mostly in maintenance mode right now, axios not too active altogether.

svozza commented 5 years ago

Just a quick note to say thank you (and all the other contributors) for all the hard work over the years on this module; it was one of the first I ever used back when I started with Node so will always have a special place in my heart.

Vivalio commented 5 years ago

Let’s just bump the major version when we deprecate. That way most people depending on the project won’t see this error until they try to upgrade to a new major, which means they are actively developing it and really should look for an alternative.

i think this is still a viable solution for the mention above.

millette commented 5 years ago

@kibertoad Looks like @mikeal is working on https://github.com/mikeal/bent. I've been using https://github.com/sindresorhus/got for many years and it's well supported and evolving.

riclf commented 5 years ago

With all this talk and the possibility of it being deprecated, I think there has to be equal mention of a current maturity replacement module, of parallel utility. We can't just announce its end and then suggest nothing, or a replacement of much less maturity and confidence. Request is used in serious applications. Why does this matter? Because for all its "outdated patterns at its core", it works on a daily basis, for thousands. This is not about the perfect world but the real world. What is the real world replacement, of confidence, on the day request is put in maintenance mode or is deprecated? That is an imperative.

reconbot commented 5 years ago

You can find that discussion over here https://github.com/request/request/issues/3143

You can find a current working plan (which direct feedback is welcome) can be found here https://github.com/request/request/issues/3142#issuecomment-478303334

aalimovs commented 5 years ago

Thanks for your work on request!

The patterns at the core of request are out of date.

Patterns change every few months and years, especially in JavaScript community. Aren't the reasons why request was originally created are still valid today?

request has 10 years of commits, stability and tests. Why start from scratch? Isn't this just adding more "JavaScript fatigue", resulting in more libraries doing the same thing - HTTP requests?

It's sad to see such an important and historic library in Node's history go away because streams and callbacks are not fancy in 2019 anymore.

stcktrce commented 5 years ago

I don't believe that deprecating the library is really needed, it's been around for about 10 years now, used in lots of places and is actually pretty stable, and in the end. all it does is make HTTP requests, what else would the library need? Support for the JS fad of the month? 👎

cboden commented 5 years ago

The committers that are still active will try to merge fixes in a timely fashion, no promises though.

ba-dum-chhh! 🥁

cphoover commented 5 years ago

This is responsible deprecation. Well communicated, with a plan to follow through on. I think other OSS maintainers can look to this as a standard to aim for.

This is much better than forgetting about a package and letting random people (who can inject back-doors into the code) in as maintainers to take over when you no longer care.

Request was a great package, and we thank you tremendously for your contributions to the early node ecosystem. You are right in your assessment that callback style is no longer idiomatic JavaScript, and there are other packages like fetch which mirror WHATWG standards.

DiegoRBaquero commented 5 years ago

@stcktrce Exactly, the library doesn't need anything else, it works just as it is. But there has been major improvements in the whole ecosystem. Deprecating the library is just marking the opportunity for others to check new and more modern libraries instead of simply relying on the most popular out there.

@mikeal thank you for all your efforts in the library (r2 too) and the ecosystem. Also, for setting this precedence of a well thought through and planned deprecation in the ecosystem.

jasonswearingen commented 5 years ago

Let’s just bump the major version when we deprecate. That way most people depending on the project won’t see this error until they try to upgrade to a new major, which means they are actively developing it and really should look for an alternative.

@mikeal I don't think that's a good idea.

The problem is that most of replacements are of lower quality than request. I just moved to request from axios about a week ago.

Axios has multi-year persistent bugs around proxy support, modifying https agents, and unhandled promise exceptions. You only find these out after investing into axios heavily.

To new users axios looks superficially as good as request (similar number of users, promises by design, etc)

jbunton-atlassian commented 5 years ago

Thanks for request :)

If anybody is looking for a minimal promise-based HTTP library with pluggable filters and good support for streams you could check out httplease. We've been using it for a few years in production.

robberfree commented 5 years ago

I love the request module.Thanks a lot. You mean request get too much focus to prevent other same new module come out?

reconbot commented 5 years ago

If there are specific bugs in comparable features in other libraries I’d like to specifically identify them. Proxy support is a complex feature and having a test case that request passes but other libraries fail is very valuable.

jasonswearingen commented 5 years ago

@reconbot in the latest axios (^0.18.0) you can't connect to a https site through a proxy server. doing so results in EPROTO errors. this is an open bug regarding this, but the issue goes back years: https://github.com/axios/axios/issues/1981

edit: specifically, you can't use axios to do https requests via a http proxy. maybe a dedicated https proxy works, didn't try that.

mivanovaxway commented 5 years ago

I sure hope fixes are not considered new features, such as my pull request for Maximum Response Size, which I see as a standard required feature of any mature library.

Also I did review other request libs before I choose this one and most of them are very problematic, incomplete and buggy. Their docs do not measure either. I do not really see what can another library bring but untested code and bugs, it's not like there's a new approach to making HTTP requests. It's all about wrapping http/https module and providing sane defaults such as buffering response, decoding responses, and of course the ability to promisify the whole thing. The biggest problem of this library here is the aim of total compatibility, trying to be compatible with legacy stuff only brings pain and legacy coding practices. But this can be fixed in many ways. There's a good base that can be refactored into something elegant, modern and minimalist. And most of all reliable. There are many ways to do this - split into more files, use ECMA6 with Babel or Typescript.

No sane developer wants 10 libraries that do the same thing but lack different features, are buggy, undocumented. This library really works and I am thankful for it and hope that it's not deprecated but instead revived.

mikeal commented 5 years ago

Fixes are not considered new features. Fixes will be merge for at least a year, possibly even longer.

-Mikeal

From: mivanovaxway notifications@github.com Sent: Thursday, April 11, 2019 2:38 AM To: request/request Cc: Mikeal Rogers; Mention Subject: Re: [request/request] Request’s Past, Present and Future (#3142)

I sure hope fixes are not considered new features, such as my pull request for Maximum Response Size, which I see as a standard required feature of any mature library.

Also I did review other request libs before I choose this one and most of them are very problematic, incomplete and buggy. Their docs do not measure either. I do not really see what can another library bring but untested code and bugs, it's not like there's a new approach to making HTTP requests. It's all about wrapping http/https module and providing sane defaults such as buffering response, decoding responses, and of course the ability to promisify the whole thing. The biggest problem of this library here is the aim of total compatibility, trying to be compatible with legacy stuff only brings pain and legacy coding practices. But this can be fixed in many ways. There's a good base that can be refactored into something elegant, modern and minimalist. And most of all reliable. There are many ways to do this - split into more files, use ECMA6 with Babel or Typescript.

No sane developer wants 10 libraries that do the same thing but lack different features, are buggy, undocumented. This library really works and I am thankful for it and hope that it's not deprecated but instead revived.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/request/request/issues/3142#issuecomment-482043697, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAACQ8I4BSRtOjqHk637gRfBhkvGbRrIks5vfwKIgaJpZM4cT_Li.

rogerkmp2 commented 5 years ago

TIL 41k packages just became vulnerable.

Look, I agree that request should go away, but I’m always fearful of mainstream packages like this changing their release pipeline. One bad actor or one compromised dev box publishing malicious code would effectively spread to every project out there.

Please consider tightening the npm push requirements. Set up a branch for ci, require multiple approvals, something more than simply pushing to master.

dooglio commented 4 years ago

no promises though.

Pun intended? 🤣

laoshaw commented 4 years ago

maybe the same logical reasoning should be applied to expressjs? for request we now have the new shiny got module, there is no re-write or true alternative to expressjs on the horizon.

express is great, but it is not really actively updated with new features these years

mikeal commented 4 years ago

express may not be updated with new features but it is actively maintained and, last time i checked, has a few people still quite interested in doing that work. i don’t know that they need to take the steps we’ve taken towards deprecation.

aalimovs commented 4 years ago

@laoshaw what has express anything to do with request?

mikeal commented 4 years ago

Preparing full deprecation. https://github.com/request/request/pull/3267

mikeal commented 4 years ago

We are fully deprecated!

All versions on npm note the deprecation and the README notes clearly that request has been deprecated.

It’s been a great 10+ years, thanks to everyone who contributed over the last decade. Let’s all look forward to new libraries that are better suited for the changes that are occurring in the JS language and ecosystem.

riclf commented 4 years ago

So lets get SPECIFIC. What is the lean code replacement for the request module?

Not to be left hanging on dead crust.... so many better options... like WHICH ones? Not GRAND do everything under the sun libraries/modules, please.

JustinBeckwith commented 4 years ago

@riclf we've been using https://github.com/googleapis/teeny-request/ to help ween us off of request for a few years. It does not do everything you are going to want it to do :) It uses node-fetch under the hood. There are other great options out there too!

jkrems commented 4 years ago

For a promise-first solution there's also gofer which is heavily focussed on API communication. Support for TCP connect timeouts built-in, easy configuration (and rich errors) for talking to multiple APIs, etc..

jeremy-j-ackso commented 4 years ago

Does anyone have any recommendations for alternative clients that have good support for HTTP Long Polling and presents as either a Stream or Event Emitter?

bajtos commented 4 years ago

The last time I checked in April 2019, alternatives like got, node-fetch and axios had a major problem: when a (low-level networking) error happened, they discarded the helpful stack trace reported by Node.js core and threw a new high-level error with a stack trace pointing to the http client library only. This made debugging transport-level issues pretty much impossible, for example when proxies are involved.

Is there any good request alternative that preserves error details provided by Node.js core?

jkrems commented 4 years ago

@bajtos I'm pretty sure gofer only decorates the original errors but should preserve the stack traces and messages.

mikeal commented 4 years ago

bent has nice errors and is designed for async/await. It’s also incredibly small and the bundle size is teeny tiny ;)

The API is nothing like request though, so I wouldn’t call it a “replacement.”

intelliot commented 4 years ago

@mikeal Why is it called bent? (request was an easier name to remember.)

Charuru commented 4 years ago

bent has nice errors and is designed for async/await. It’s also incredibly small and the bundle size is teeny tiny ;)

The API is nothing like request though, so I wouldn’t call it a “replacement.”

This feels very much like technically correct rather than user-friendly logic. From the user perspective bent solves the same problem as request but better. Now it's stuck with a worse name for no reason. You can call it request 3 without much issue imo. Yes the API is breaking but what's what we have semver for.

mikeal commented 4 years ago

You can call it request 3 without much issue imo. Yes the API is breaking but what's what we have semver for.

Spend some time with bent and you may feel differently.

It’s not a small difference in naming or promises vs callbacks. The ergonomics are very different, the states it surfaces are very different, the way it thinks about error conditions is a radically different approach.

request is a more procedural API, you tell it to do something and it tells you what happened, it only gives an error if something irrecoverably failed. bent takes the success criteria for the entire lifecycle and returns you an API that will fail if anything but the success criteria is met.

You use these libraries very differently. There are other libraries that are closer to request’s API if that is what you want, but after almost 20 years of working on HTTP clients I’ve found a different and ultimately better approach that I’d encourage people to consider, but I’m not going to ram it down everyone’s throats by making it request 3.0.

Why is it called bent? (request was an easier name to remember.)

Because you “bend” it into a specific shape (very particular success criteria) and it provides an ideal API for the success of that shape and fails on anything but it.

The name is a bit abstract, but request is the kind of name you could never get today. I barely got request in the npm registry and I wrote the original npm registry 😜

laoshaw commented 4 years ago

what about "got" as a replacement, it's sad we don't have a clear replacement while request is officially deprecated.

mikeal commented 4 years ago

what about "got" as a replacement, it's sad we don't have a clear replacement while request is officially deprecated.

Maybe we should take the fact that nobody has written an API compatible replacement as an indication that adopting an API compatible replacement is undesirable once you sit down and work on it 🧐

That was certainly my experience.

psequeirag commented 4 years ago

Perhaps what people really want when they ask for a "replacement", is not so much an API compatible alternative, but the maintainer perspective on what other packages are already out there to solve roughly the same problem and that are making this package irrelevant so that you can confidently call it "deprecated".