deprecate domains

[jonathanong]

what's the status on domains? can we deprecate it before v1?

[scottnonnenberg]

@marak I have a really hard time with this line of thinking. It seems like you've got some sort of Javascript/async Stockholm Syndrome. "You mess up with async, you deserve it! We don't deserve better!"

I can't accept your item number 2 or 3 above, since it doesn't allow us to associate the error with the incoming request that caused it. This is important both for proper debugging and for surfacing some sort of error back to the original client request which caused the error. And number one is obviously impossible.

Additionally, I find it interesting that you consider domains to be a complication, when for client code they simplify everything greatly. hapi users don't need to their why their server stays up after an error. Most low-level libraries implement domains properly at this point, so the average developer doesn't have to think about it at all. And their applications aren't any harder to reason about.

[Marak]

since it doesn't allow us to associate the error with the incoming request that caused it

I've literally never needed to know this information. In the rare case where I need to run unreliable code and actually want to show the stack trace to the client, I simply use the third item I described above ( which is spawning a child process, letting it die, and piping stderr to client )

hapi users don't need to their why their server stays up after an error

Servers should not stay up after an uncaught error. They should die and restart immediately. An uncaught error by definition means that something unexpected happening in your application code. The moment something unexpected happens in your application you have entered an indeterminate state. Trying to reason about something in an indeterminate state is by definition difficult.

[scottnonnenberg]

@marak 'unreliable code' is the same thing as 'all code.' Therefore, I do run multiple instances of my servers so one can go down an the others will continue to run. And I agree with you that servers should go down after an error.

But I also think that anything in-process in that server at the moment should be completed before that happens. We're not talking about toy apps here, we're talking about large-scale apps with many connected clients and in-process tasks spanning multiple data stores and third-party APIs. Yes, really bad things will happen (like power outages) but we shouldn't allow destructive things to happen any more often than they absolutely must.

[Marak]

@scottnonnenberg -

Why exactly are your applications crashing in the first place? Is there a classification of error which is frequent? Are you dealing with multiple contributors and insufficient test coverage?

Simply speaking, the only time I've ever encountered actual uncaught errors in high-traffic node.js applications has been around things like calling res.writeHeader with invalid data. Was very easy to identify and fix without much debugging.

I don't need tasks to complete in my applications. If any task is important enough that completion must be guaranteed, then what I really need is acknowledgment receipts with a distributed consensus ( since the task could complete, but power-out on sending acknowledgment ).

If you are interested in which syndromes I may or may not have, maybe try taking a read at this article:

https://hook.io/blog/the-monolith-versus-the-microservice-a-tale-of-two-applications

My general attitude is not You mess up with async, you deserve it!, but more like, We are all going to mess it up. Better to isolate the problem in a separate process.

[ofrobots]

Folks, lets keep this issue for discussion on deprecation of domains. This thread should not be used to debate over merits of different application architectures.

[Fishrock123]

Most people haven given up on replacing/using domains at this point. Generally it's an unreasonable amount of work. Anything like this ties so deeply into everything it is almost impossible to maintain well. Also coming up with the correct API and error handling heuristics is extremely difficult.

Perhaps there will be an alternative in some future, but don't count on it. Please do not use domains unless you absolutely need them.

So in the time between my last comment, @trevnorris has been working a good deal on something known as AsyncWrap.

Domains are a huge headache for us to maintain. Subtle edge cases abound which are difficult to fix, and exceedingly hard to reason about.

While simpler domain use cases may be nice, working magic, those edge cases are a nasty minefield and only make things worse for everyone.

In light of this, AsyncWrap has been a long time in the coming. Base parts have already been implemented for a considerable amount of time but nailing down a good JS API for AsyncWrap is important, which is why it's taken quite some time.

While AsyncWrap is not a drop-in replacement for domains, we're very confident it will be able to handle more cases, better, and also be far less of a burden for us to maintain.

"AsyncWrap dictates how things will work. whereas domain has to consume how everything else has been built." — @trevnorris

Note that the timeframe for which domains may possibly be removed is not within sight yet and they will continue to work. We'll update everyone once more progress has been made. :)

For now however, the official recommendation on domains will continue to be the same: Please do not use domains unless you absolutely need them.

(Also for anyone who asks about Zones: AsyncWrap pre-dates them, and is likely able to handle more use-cases.)

[Fishrock123]

(Also please don't post +1 / -1 etc comments or things that have already been discussed above, I'd rather not have to remove comments or lock this. Thanks!)

[DaSpawn]

This appears to come down to personal preference/perceived place in node rather than specific need to remove it for a security or functionality reason.

@Marak you make excellent points, but change the perspective; I need a safety net from the end user input, not an error handler:

1. Domains should not be used specifically for error handling overall, that would appear to be a lazy/bad way to use them, and should still fall under process handler. My code should never do anything I do not expect it to, so I still have error handling for specific cases as needed/expected. What I can not guarantee however is user input. If a user uploads a file that is somehow unexpectedly incompatible, now or in the future, I do not want that to crash the process, there is no reason to, what we need to do is gracefully exit without affecting any other current user requests/actions, which could also be in the middle of processing a task and have no problems, and it does not matter what the task is, it is all simply bubble wrapped in a domain as needed
2. global exception handler should be simple and never change, it should be the all else has failed and ensure exit and cleanup handler that never gets called. The possibilities of problems arising from changing the handler randomly are limitless, plus, you should never mess with objects above you unless you are supposed to (top down)
3. unreliable code is not the purpose/need for domains. I already make use of child processes to handle requests, but the code is stable and error free, the user input is not. I need to ensure one users actions do not kill a web server process just because they inadvertently uploaded the wrong file and abort other requests in the process at the same time. Child processes are extremely slow to fork, as expected, and would make a site crawl if I had to fork every request just so I have simple request isolation to prevent unexpected errors affecting anyone but the user that caused it. I can maintain the same task isolation I need with domains without child processes. Child processes also add much complication to a project just by themselves, and unless you have linux background/knowledge they can be easily implemented wrong in scale

domains are the perfect fit for many situations where you want to ensure task containment but not for overall error handling, but how people use them is their choice. Domains is very simple small amount of code, and part of the core of node, ie they require no other modules but core modules to function, and their simplicity ensures that complications in the domain implementation is not the creator of the error, so adding/needing promises or other large modules at the top of code structure could be more problematic. Domains follow the core principle of node, callbacks

This is not about ensuring node stays like other functional languages, this is about ensuring node gives the greatest flexibility in functions for it's users needs. Domains ads a perfect simple dimension to an application that gives much greater confidence in application stability and uptime that would require adding other complicated modules and methods to achieve, and as @ofrobots said this discussion is not about this, just domains themselves

@Fishrock123 AsyncWrap sounds interesting, but that was not the use case here and ads much more complications/code, all I needed was a safety net and domains provides it perfectly. What is the edge cases you are talking about that makes it difficult to maintain? AsyncWrap sounds like it could be a better replacement in those cases though.

Just having domains listed as depreciated means that very simple built in functionality could disappear at any time, leaving usage 100% unreliable for future planning, even if domains stays forever

[algesten]

@DaSpawn the counter argument to your point has already been made below. in short: no one is challenging the usefulness of domains. however the technical debt/consequences throughout core are too much to keep them.

[Fishrock123]

Domains ads a perfect simple dimension to an application

Right, it's a magical poison chalice.

Domains is very simple small amount of code

This is most definitely not true.

What is the edge cases you are talking about

There's some info here on AsyncWrap vs Domain vs Zones: https://github.com/angular/zone.js/pull/231#issuecomment-176899725

that makes it difficult to maintain?

See my above post: "AsyncWrap dictates how things will work. whereas domain has to consume how everything else has been built." Domains tie into absolutely everything in the worst ways.

Here is a domains bug no-one knows how to fix because it makes no sense or reason at all: https://github.com/nodejs/node/issues/1271

Just having domains listed as depreciated means that very simple built in functionality could disappear at any time, leaving usage 100% unreliable for future planning, even if domains stays forever

I just said above we won't remove it until a good while after AsyncWrap is in. Please be sure to re-read my above post as I cover a lot of things I don't feel like having this thread go much longer than it already is.

AsyncWrap sounds interesting, but that was not the use case here and ads much more complications/code, all I needed was a safety net and domains provides it perfectly.

You should still be able to implement the same thing in AsyncWrap, and it will probably be more reliable.

[Fishrock123]

(Also to be clear: The decision has already been made long ago that we are moving away from domains.)

[DaSpawn]

Thank you for taking the time to clarify everything, lots of good information; hopefully a solid replacement recommendation (like AsyncWrap) can replace/compliment the long term depreciated page/flag

[ORESoftware]

I am not sure I understand, any replacement for domains would basically have the same behavior as domains so why replace it?

[algesten]

@ORESoftware

the technical debt/consequences throughout core are too much to keep them.

[ORESoftware]

@algesten I think you mean that domains require way too much global state and are therefore way too expensive to maintain, which makes more sense than what you just said? I would buy the argument that I just made, but we need something like domains, the functionality is critical. I haven't checked out the alternatives to domains, but the only good alternative is one that somehow manages to avoid global state somehow, otherwise it's the same poison.

[benjamingr]

@ORESoftware your promises example would not suppress the error. You need to add a rejected promise handler - https://nodejs.org/api/process.html#process_event_unhandledrejection

[ORESoftware]

@benjamingr thanks that's actually some pretty useful info - I think it's on the wrong thread though

[Fishrock123]

I am not sure I understand, any replacement for domains would basically have the same behavior as domains so why replace it?

Not necessarily. Please see my above comments for more of a write up. AsyncWrap is capable of achieving the same goals, at almost no cost to us. The API and behavior of AsyncWrap itself will not be identical, but you would be able to make a domains-esque module with it, that should cover all the same use-cases.

[ORESoftware]

@Fishrock123 nah, from what I have read, I don't think asyncwrap can do what domains can do. domains provided the possibility of better stack traces and universal error handling for the entire node.js ecosystem. AsyncWrap looks like it might be only useful for applications, where applications can protect themselves from their own code but it cannot protect applications from unexpected errors thrown in other libraries nor can AsyncWrap allow for development of test frameworks where errors are expected to be thrown from asynchronous code. The reason why I am fairly well familiar with the issue at hand is because I am trying to develop a test framework and I am realizing it is impossible to create high quality software in Node.js without 100% domain support. See this issue in Ava, an up and coming testing library that surely will I believe be thrown for a loop by the issue of domains no longer being supported

https://github.com/sindresorhus/ava/issues/567

I repeat, you cannot create a solid asynchronous testing library for node without something very much like domains and asyncwrap and promises do not solve the problem, not even close.

[DaSpawn]

@ORESoftware this is the same wall/conclusion I arrived at also as I had implemented domains in a multi-fauceted dev project and it works perfectly as expected/needed to isolate everything and still catch the errors that are expected/needed to debug quickly. I came to node docs to get more domain info, then seen the depreciated and arrived just where you are also

Node has stated that this decision was decided long ago to depreciate domains (not going to just remove domains without something that can cover the use-cases though), and there is nothing we can do about it really. Honestly this makes absolutely no sense to me as there is significant need and use for domains we both have arrived at in creating enterprise grade software, but they do not see these use cases and insist the functionality should be in another module since too many people use domains exactly as they should not, expect them to work, then complain when their projects break due to issues already known with domains

[Fishrock123]

I'm pretty sure this has been stated before, but: we are not going to just remove domains without something that can cover the use-cases.

[ORESoftware]

@Fishrock123 well, first of all, Node core did "remove" domains because they aren't supported by ES6 native promises, so that's already a problem for me. There are always use cases that are beyond the comprehension of a single person, but hopefully not all of Node TC. here is what you might not understand - the only thing that can do what domains do is domains. Or something exactly like them, different in name only. They are a clever solution to a difficult problem and currently I believe the only solution to that problem. Any other solution will not cover all of the use cases that domains cover. The use case that I have, as I said, is creating a test framework that can run tests asynchronously (in "parallel") in the same Node process. What this means is that at the global scope I really don't know for certain which test is running at any moment, because of asynchronous callbacks that could be invoked at any time. So if errors bubble up to 'uncaughtException' (which was at one point deprecated by the way, but I guess they backtracked on that) or 'unhandledRejection', it's already too late, I cannot pin the error to a specific test. Domains, however, solve my problem perfectly. I am sure supporting domains is difficult, but I think it's way worth it.

[rvagg]

If you have a usecase for Domains where they are currently broken, native Promises for instance, then you are welcome to work up a pull request to fix it how you consider appropriate and we can have that discussion then.

There is currently nearly zero interest amongst the core team for maintaining promises so complaining here will get you nowhere but this is how open source works. Contribute something acceptable and you'll be welcomed with open arms, in fact it'd probably be nice to have someone working in core who cares about domains (rather than actively hating the difficulties they cause like many of the team). Just keep in mind that anything beyond strictly "fixing" domains, such as adding new functionality to domains, is not likely to get anywhere either due to the deprecation status.

It might also be worthwhile investigating the potential that AsyncWrap has for building solutions to the same kinds of problems that domains were attempting to address. You may find that use cases are already solved in fact. The API hasn't been finalised or formally approved but it exists undocumented in v4 and v5 if you want to start trying it out and contributing to its evolution.

[ORESoftware]

@rvagg thanks, and thanks to @Fishrock123 for the discussion - I want to patch ES6 promises so that they work with domains, I presume the location for native promises is here

https://github.com/v8/v8/blob/40d3095c2e16106f122db744a29a9e1a804442ce/src/js/promise.js

looks pretty nasty to me, not sure I am capable of creating a quality patch, TBH, but I will give it a go tomorrow

I am not against AsyncWrap if it can solve the problems I have now that are only solvable with domains, but right now I am simply doubtful on that.

[RobertWHurst]

This is an interesting discussion. I want to add my own input and see what you guys think. I think it's important to keep things manageable inside node core, and I can imagine domains are a real pain to implement, but I don't see an alternative to them. Firstly I think I'm in alignment with the 3 points that @DaSpawn laid out above. Additionally I think try catches in every callback, or using promises and generators doesn't seem like a sane solution, in fact I believe it's just pushing the complexity into the hands of the engineers who use node. The whole point of node as I understand it is to make creating async servers less painful. Ryan achieved that goal by forcing the program to run in an event loop. One of the consequences of the event loop model is that you loose the call stack, including try catches, upon each tick. I think that is an acceptable trade off as the expressiveness gained when dealing with concurrent logic is very powerful. That all said, domains make the negatives of this trade off a lot less painful. Cordoning off subsystems within a process is a good way to manage uncaught exceptions. Those of us who author web servers or frameworks should be able to handle internal errors independent of the rest of the code running within the same process. With only one global catch all for uncaught exceptions you loose the ability for these systems to recover independent of the the rest. The separation of concerns is lost. I think errors within http handlers is but one example of this.

I'm looking forward to how you guys improve this solution, but I personally insist that domains or something similar need to exist in some form. No one ever writes perfect code, and everyone has triggered a runtime error at some point in their development experience using node. The expectation that engineers must write perfect code is unreasonable. To my knowledge no other modern language expects that of users. I feel domains, or something similar solve this issue. Without them error handling is seriously undermined.

[jondubois]

Domains are useful for frameworks - They're a good way to separate 'user logic' from internal 'framework logic' for the purpose of failure handling and recovery.

Note that 'failure handling' is different from 'error handling' - A failure is a side effect of an Error - The user of a framework should be allowed to handle errors however they like, but sometimes you want the framework to decide how to handle the subsequent failure which resulted from an error (perhaps even if the error was caused by the user logic).

For example, the framework itself might invoke a user-defined function (for example, middleware) which might branch off (in unpredictable ways) and invoke various functions synchronously and/or asynchronously - Under such circumstances, if the user's code throws an Error, you may not want to just kill the whole server (especially when it comes to WebSockets because then you would end up killing tens of thousands of perfectly good connections along with it) - You just want to shut down the single socket which was related to the problem.

Basically, as a framework developer, domains offer you the ability to control the failure handling (and recovery) aspects around code which was written by a third party.

A big practical problem for example is if there is a logic error in user code - For example, if the user code tries to invoke a function which doesn't exist - You DO NOT want the server to crash (and take down all other sockets) - You just want to kill a single socket.

If the error happens within the app's core logic (not related to a specific socket or request, but to the system as a whole), then in that case, you may want to bring down the whole server.

By removing domains, we may be forcing users of frameworks to devise their own 'failure recovery' procedures which would otherwise have been handled seamlessly by the framework itself (in the most optimum way).

EDIT Upon further consideration, it appears that Promises offer similar features as domains when it comes to failure containment. Anyone tried this approach? For example, you can just add a .catch(handler) to a promise which is wrapped around the user logic.

In any case, I think this approach should be investigated thoroughly (and documented) before domains are removed completely.

[ORESoftware]

100% agree with that assessment. As a test framework developer where user code calls the framework api, domains are an imperative. In the case of test framework, errors/exceptions are supposed to happen and be handled without explicit use of try/catch.

On May 4, 2016 6:13 AM, "Jonathan Gros-Dubois" notifications@github.com wrote:

Domains are useful for frameworks - They're a good way to separate 'user logic' from internal 'framework logic' for the purposes of failure handling and recovery.

Note that 'failure handling' is different from 'error handling' - A failure is a side effect to an Error - The user of a framework should be allowed to handle errors however they like, but sometimes you want the framework to decide how to handle the subsequent failure which resulted from that error.

For example, the framework itself might invoke a user-defined function (for example, middleware) which might branch off (in unpredictable ways) and invoke various functions synchronously and/or asynchronously - In such a situation, if the user's code throws an Error, you may not want to just kill the whole server (especially when it comes to WebSockets because then you would end up killing tens of thousands of perfectly good connections along with it) - You just want to shut down the single socket which was related to the specific action which caused the problem.

Basically, as a framework developer, domains offer you the ability to control the failure handling (and recovery) aspects related to code written by a third party.

A big practical problem for example is if there is a logic error in user code - For example, if they try to invoke a function which doesn't exist - You DO NOT want the server to crash (and take down all other sockets) - You just want to kill a single socket. (That would classify as a security vulnerability).

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/nodejs/node/issues/66#issuecomment-216857830

[jondubois]

The other really nice feature which domains offer is the ability to listen to the 'error' event on an EventEmitter without explicitly binding a listener to its 'error' event. E.g. using domain.add(emitter).

This relates to a common use-case for frameworks - You don't want to allow the user to accidentally unbind 'error' handlers which were setup by the framework for its own internal use (e.g. if the user calls eventEmitter.removeAllListeners()) - Because that would mess everything up internally (the user should not be able to unbind error handlers which they did not attach themselves).

[ORESoftware]

does anyone know if there is indeed an effort to patch native ES6 promises to work with domains? Will that land in a Node version soon?

[trevnorris]

@ORESoftware Either we would need to overwrite native Promise, or use a V8 provided API to propagate the information. The later is slow going and won't happen soon, and doubt the CTC would be willing to patch Promise directly.

[ORESoftware]

@trevnorris ok thanks for that info.

[jondubois]

@trevnorris

Instead of messing with native Promises, couldn't we just rewrite the domain module itself to leverage Promises instead? It looks like doing this would allow it to automatically work with nested Promises and also with async operations like setInterval, nextTick etc...

So for example, the domain.run(fn) method could just be:

Domain.prototype.run = function (callback) {
  var self = this;

  (new Promise(function (resolve, reject) {
    var args = Array.prototype.slice.call(arguments, 1);
    callback.apply(self, args);
  })).catch((err) => {
    this.emit('error', err);
  });
};

I tried this in a number of async scenarios and it seems to behave the same as with the Node.js domain module's run(fn) method - It's actually better because unlike the domain module, this doesn't pollute Error objects with error-domain-related properties... The promises do everything magically and capture any async errors.

I guess it doesn't capture emitted 'error' events though...

[bayov]

I haven't read the whole thread, so sorry if I'm repeating things.

I want to say that I've recently been using domains to capture all calls to console.log, console.warn, etc. It is not a simple matter of monkey-patching these methods, since I also need to separate each call to its source (which is a unique context defined by my application).

So, I'm not using domains for handling errors at all. I don't even register an error event listener on my domains. I'm using domains as a way of execution context management.

This feature is necessary for my application to work, and so I hope that any replacement API to Domain will be able to cover my use-case...

BTW, I would be glad to know if there's a better way to do what I'm trying to do, other than abusing domains.

EDIT I just wanted to add that it's not possible for me to improve my application to keep track of context manually, as my application keeps track of third-party user-code.

---

N.B. I've also encountered the issues with native Promises. In my case, I fixed it by monkey-patching Promise (even though it might have a big performance hit):

function deferResult(fn, ...args) {
    let hasError = false;
    let error;
    let result;

    try {
        result = fn(...args);
    } catch (err) {
        hasError = true;
        error = err;
    }

    return () => {
        if (hasError) { throw error; }
        return result;
    };
}

const originalThen = Promise.prototype.then;
Promise.prototype.then = function (onFulfilled, onRejected) {
    const activeDomain = domain.active;

    let onF = onFulfilled;
    if (activeDomain && typeof onFulfilled === 'function') {
        onF = function (value) {
            activeDomain.enter();
            const deferredResult = deferResult(onFulfilled, value);
            activeDomain.exit();
            return deferredResult();
        };
    }

    let onR = onRejected;
    if (activeDomain && typeof onRejected === 'function') {
        onR = function (reason) {
            activeDomain.enter();
            const deferredResult = deferResult(onRejected, reason);
            activeDomain.exit();
            return deferredResult();
        };
    }

    Reflect.apply(originalThen, this, [onF, onR]);
};

[ORESoftware]

@sosz please search and read the whole thread before, I patched promises to use domains myself, in a much simpler way

Your code is a little bit of a horror show. What about this instead?

const then = Promise.prototype.then;
Promise.prototype.then = function (fn1, fn2) {

  if (process.domain) {
    fn1 = fn1 && process.domain.bind(fn1);
    fn2 = fn2 && process.domain.bind(fn2);
  }

  return then.call(this, fn1, fn2);
};

Note that you do not need to patch the Promise constructor, because the executors run synchronously, but you can patch Promise.prototype.catch to work in the same way as the above patch on Promise.prototype.then. That being said, pretty certain Node version 8 has allowed domains to work with Promises, so the above patch may only be useful for Node.js versions < 8.

[ORESoftware]

I just want to add my 2 cents to the long run discussion. I have a job. I want to keep my job. I really prefer to not have my server(s) crash, even if we use forever or supervisor to restart upon crashes.

Using the following allows us to avoid many crashes:

let domain = require('domain');

app.use(function(req, res, next){

  let d = domain.create(); // create a new domain for this request

  res.once('finish', function(){
    d.exit();
    d.removeAllListeners();
  });

  d.once('error', function(e){
    // log the issue right here
    if(!res.headersSent){
        res.json({
          error: e.stack || e
       });
    }
  });

  d.run(next);   // we invoke the next middleware
});

we use domains in production. Hopefully no memory leaks though 😂

[evanlucas]

@ORESoftware I've moderated your previous comment. I found it to be inappropriate for this issue tracker. We intend on keeping this issue tracker professional and would appreciate if everyone who posts here does the same.

[nickserv]

Domains are still a hack, letting the process crash is often the better option especially with forever/supervisor.

[ORESoftware]

@evanlucas censorship always appreciated, but I guess you're right, best to keep politics off of Github lol...@nickmmcurdy - you'd have to back up that statement - domains will fail in their intention if an error is raised for the wrong path of execution, and if it's easy to misuse them than that's also bad. But I don't think they are a hack perse, just perhaps imperfectly implemented.

[addaleax]

@ORESoftware We have a Code of Conduct that asks you to be respectful of other people’s viewpoints and refrain from using language that “could reasonably be considered inappropriate in a professional setting”.

If you don’t feel comfortable with that, it might be easier to take a step back; Node’s project leadership is standing behind the choice to apply that CoC to our spaces.

[ORESoftware]

Imma read the CoC right now to see if my microaggression actually violated it.

[ORESoftware]

Yep, looks like I violated point 1

• Using welcoming and inclusive language

whoops, my bad fam