What is your biggest pain point w/ Node?

[mikeal]

What is the biggest pain point right now using Node?

This doesn't have to be specifically related to Node core.

Please refrain from suggesting features yet.

Please include a little bit about your background and what you're doing with Node.

[trevnorris]

There's too much surface area to the API that's tightly coupled with the internals. By writing a lower level binding (what @domenic has dubbed "libuv.js") that much of the current API can be implemented on top of, Node can allow developers to implement their own interfaces.

There's sentiment from one group that Node should have full support for Promises. While at the same time another group wants generator syntax support (e.g. var f = yield fs.stat(...)). It seems these parties miss the fact Node can't simultaneously support all syntaxes for all upcoming features.

That's why I believe core code needs to minimize the API and allow these parties to implement the interface of choice. These are easily enough installed via a module, and it reduces the surface area core maintainers have to maintain. This will lead to more secure and stable code from core, while also allowing interface implementors to 1) iterate faster than core releases occur; 2) bypass the current overhead of needing to implement any other interface on top of the current EventEmitter/Streams model.

[trevnorris]

Heh, seems I didn't make it to point #3 in the OP.

[jonathanong]

btw generator support === promises. if they ask for something different, they don't know what they're talking about.

i'd be pretty interested in libuv.js.

my biggest issue is right now is that it seems as though the node lead only spends a single day out of the month to work on node.js ~_~

[bodokaiser]

I think the biggest problem with node is that it lost a lot dynamic. Just look at the release cycle: We are more or less waiting (or using) 0.11 for a year now and for me it does not look as 0.12 will be released 2014. I understand that the core team wants to ship a product enterprise ready by freezing apis and rejecting any major changes but in the long term a lot of people will loose interest as PRs get rejected and bad api decisions maintained.

Regarding libuv.js I see potential to decouple dependency from the nodes api decision but am not sure if the community is able to support various architectures in terms of documentation and reliability.

[feross]

I agree with @jonathanong and @bodokaiser - node 0.12 is way past overdue. The goal of shipping an "enterprise ready" product is laudable, but having 0.12 be in a perpetual "right about to be released" state actually hurts credibility way more than releasing 0.11 right now as 0.12 would.

[juliangruber]

I'm listing my 2/3 points without thinking about whether they make sense to be fixed in node:

• not having several call stacks / threads: you can't just let a block of execution crash, knowing safely that you don't need to crash the process. like: each http request is completely independent and a crash like an uncaught "error" event somewhere wouldn't affect the other requests
• versioning of node core modules, i want readable-stream style versioning for every core module. that would also help with release cycles
• too long release cycles, when I need change X i need to use unstable node

[Qard]

I'd like to be able to use es6 tech with core.

Streams could be simpler:

var upcaseStream = passthroughStream(function* (read, write) {
  var bytes = 1024
  var chunk
  while ((chunk = yield read(bytes))) {
    yield write(chunk.toString().toUpperCase())
  }
})

process.stdin.pipe(upcaseStream).pipe(process.stdout)

Servers could be simpler:

// textual interface (get foo, set foo bar)
tcp.createServer(function* (read, write) {
  var auth = yield read()
  if ( ! (yield db.auth(auth))) {
    return
  }

  var message
  while ((message = yield read())) {
    var match = message.match(/^(get|set) (\S*)( (\S*))?$/)
    var command = match[1]
    var key = match[2]
    var val = match[4]

    if (match) {
      yield write(yield db[command](key, val))
    }
  }
})

At the same time, I agree with @trevnorris that this sort of concern of promises vs generators should probably not be directly in core. However, having some of these features enabled by default in V8 would promote wider acceptance of their use--I think co/koa would be a lot more popular if you didn't need to use a special flag on an unstable build.

I really like the libuv.js idea. As someone whose job is instrumenting node, having deeper access into the guts of it would be really valuable. I'd really like to see visibility into uv_check_t, uv_prepare_t and uv_idle_t at JS level.

Having something like Zones in core would be great. Perhaps with libuv.js we could store file descriptor references directly on JS objects so cleaning up file descriptors of things in a zone could be as simple as socket.fd.destroy().

I'd also love to see core facilities designed in a way that they are more pluggable. For example, with browserify you can use transforms to make require('./path/to/template.jade') return a compiled jade template function. Also, monkey-patching require() behaviour is currently rather difficult to do while retaining correct path resolution--it'd be good if there was a way around that.

My biggest gripe, however, and the reasoning for my strong desire for generator-based streams is the fragility of event emitters propagating error events. Having to special case error events is kind of obnoxious. With generators, you can simply let the user try/catch it, or leave it to throw in normal JS fashion.

Cleanup also kind of sucks. If you forget to end a stream, bad stuff can happen. With the generator example above, you can simply assume that reaching the end of the generator function means the stream is done, since there is no longer code to operate on it.

[rvagg]

@Qard your streams suggestion is absolutely something that should be experimented with in user-land in order to prove its utility. Major departures like this should never again be dumped straight into core without having a long burn-in time in user-land where they can be embraced or rejected by a wider group.

Easy to put together a wrapper, then promote it and see if you can get any traction. Personally I'd like to see core streams stripped back even further to something like what http client is now such that its a logical choice to add an abstraction over the top of it to get real work done. Then we avoid lock-in, complexity in core, and allow the ecosystem to come up with the best patterns rather than having it forced on them.

[Qard]

@rvagg Yep, I totally agree that is a userland thing. I feel like the need for a harmony flag and unstable build currently is holding people back a bit from experimenting with it though. I'm more just advocating for at least a stable release at some point with the harmony flags, though I'd prefer to see some features on by default.

As @mikeal put it, I'm not suggesting we start implementing my pet feature ideas. I'm just sharing my views. I do think the whole error event thing still kind of sucks though. Generators may be a possible solution, but, as you say, it's yet to be seen as the solution.

In line with what you are saying, and the libuv.js discussion, stripping core down even further would be really cool. I think sometimes core does too much. Having lower-level control of things would be great.

[rvagg]

Agreed on the special-case 'error' event, I just don't see a good solution to that in Promises or Generators yet, but I'll try and reserve too much judgement. In short, Promises treats errors like a second-class citizen which is exactly the 'error' event problem so it buys you nothing, Generators takes us back to codebases littered with try/catch which I certainly don't want to see become an idiomatic Node style. But, I look forward to being convinced of a great new pattern that someone clever comes up with to solve all our woes.

[Qard]

I certainly prefer manual try/catch to promises just eating errors. It is a bit verbose though. A rescue expression, like ruby, would be really nice.

[calvinmetcalf]

@Qard https://github.com/calvinmetcalf/es6-stream

[Qard]

Nice. I actually have an implementation of my own half done. I'll probably finish it on the long bus ride I'll be on tonight. Mine includes readable and writeable streams too. :)

[calvinmetcalf]

Figured you were writing one but couldn't help myself.

I mean writable is easy you just do thunkify with the native .write method, readable could be trickeir, but a thunk version of noms (which is pretty much how I wish streams worked) might be a starting point.

edit: spell my name right

[calvinmetcalf]

on second though I know exactly what you mean by read and write streams because I wrote them to make that library work and can just separate them out

[Fishrock123]

I think most of my pain points actually come from npm. Additionally, if we could get node-forward to eventually release a npm-fixed 0.8, that would be great.

Aside from that, faster releases. I know way too many people running 0.11 in production for generator stuff haha.

[trevnorris]

@Qard Node won't be enabling any harmony flags by default. We leave it up to V8 to tell us when those features are "ready". Though, if it's any consolation, we'll probably upgrade to v3.29 before v0.12 is released. This removes generators from behind the harmony flag.

As for your streams implementation and comments about the event emitter, I agree that the current implementation is not as it should be. Though no one can agree on the Right Way. Hence the need for a lower level API to allow module authors to implement their own interfaces.

Whether devs like it or not the basic callback style is the fastest way for JS and C++ to interact, and even using that we have too much overhead.

[joeybaker]

My biggest pain point is debugging.

1. Errors suck. They're a mixture of unclear and infrequently useful in an async world (because the stack traces get broken). Domains weren't the answer. I've heard a rumor that async stack traces are coming in 0.12 but… I don't think I've see it in 0.11 yet?
2. "dev tools" are lacking. Things like node-inspector are close, but I want call stack flame graphs and memory profiling. I don't want to learn dtrace if I don't have to.

IMHO

About me

I've been working with JS professionally for 5+ years now and node for about 2.5. I like callbacks, streams, tape tests, and hiking. Oh, and beer. :beer:

[Qard]

@trevornorris Tracking latest V8 versions is all I hope for in terms of es6 features. Node has lagged behind quite a bit lately.

I agree that the generator implementation may not be the right way to do streams. It has some obvious advantages, but may also have subtle disadvantages. JIT optimization performance is as-yet unknown.

A lower-level interface for streams would be great. I'd like to see the current evented design abstracting something deeper and more manual.

[rvagg]

This discussion on streams is exactly the kind of discussion that I'd like to see moved out of core and into a more focused group around readable-stream so we don't have API design being driven by core implementation concerns but the other way around.

[trevnorris]

@joeybaker

I've heard a rumor that async stack traces are coming in 0.12

Sort of. That was part of the async listener patch. Though the user-facing API has been scrapped for further iteration. There is still a "less public" API user can tap into, and we hope the will so we can get feedback on the best approach.

I want call stack flame graphs and memory profiling

There's definitely more we can do, but asking for a flame graph via a function call is a bit of a stretch. :)

@Qard

Node has lagged behind [v8] quite a bit lately.

Main reason for the lag of V8 updates is because in v3.26 V8 removed the internal debugger. Which meant we needed to implement our own. Took a while but @indutny stepped up and got it working.

JIT optimization performance is as-yet unknown [for generators].

It's an unknown because generators can't be optimized. :-P Like, straight up are never sent to the optimizing compiler. Maybe one day they will be, but as of right now there is no question of whether they are slower.

@rvagg

This discussion on streams is exactly the kind of discussion that I'd like to see moved out of core

I agree, hence why I want to create a non-streams, callbacks only, based interface for the lower-level API that everyone can use to implement their own streams interface.

[bodokaiser]

+1 for callbacks in core +1 for streams out of core - most implementations are anyway out of stack

Am 14.10.2014 um 01:46 schrieb Trevor Norris notifications@github.com:

@joeybaker

I've heard a rumor that async stack traces are coming in 0.12

Sort of. That was part of the async listener patch. Though the user-facing API has been scrapped for further iteration. There is still a "less public" API user can tap into, and we hope the will so we can get feedback on the best approach.

I want call stack flame graphs and memory profiling

There's definitely more we can do, but asking for a flame graph via a function call is a bit of a stretch. :)

@Qard

Node has lagged behind [v8] quite a bit lately.

Main reason for the lag of V8 updates is because in v3.26 V8 removed the internal debugger. Which meant we needed to implement our own. Took a while but @indutny stepped up and got it working.

JIT optimization performance is as-yet unknown [for generators].

It's an unknown because generators can't be optimized. :-P Like, straight up are never sent to the optimizing compiler. Maybe one day they will be, but as of right now there is no question of whether they are slower.

@rvagg

This discussion on streams is exactly the kind of discussion that I'd like to see moved out of core

I agree, hence why I want to create a non-streams, callbacks only, based interface for the lower-level API that everyone can use to implement their own streams interface.

— Reply to this email directly or view it on GitHub.

[Qard]

Yep. I was aware the debugger was gone.

Good to know about the generator performance. I haven't taken the time yet to dig into V8 internals to see how the new stuff really works.

[Fishrock123]

It's an unknown because generators can't be optimized. :-P

I still wonder if regenerator doesn't actually make faster code.

[joeybaker]

@trevnorris Yea, I don't expect a flame graph to be a function call away. I do think it should be easier to get a CPU profile out of V8 from node though :)

I'll go back to my main point though: ease-of-use is one the key factors in picking a platform. I'd define that as:

1. easy to maintain:
   • [ ] backward-compatible core releases: we'll get there.
   • [ ] actionable errors: core errors are pretty opaque
   • [x] stability: spin up a node process. It doesn't die easily.
2. ramping up new hires
   • [ ] easy to grok new code: JS has no standard format, has many ways to do the same thing, and has no common framework like ruby. There are pros/cons to each of these, but it does make new code harder to quickly understand.
   • [x] easy to understand the language: JS async is still a mindbender for beginners which we could make better, but JS is pretty good, and that's a JS problem, not a node problem. Also: not erlang.
   • [ ] good docs: they're somewhere between barely-sufficient to a mess
3. open source support (combined with the quality of that code)
   • [x] have something like npm: we have npm
   • [x] packages are generally of high quality: thank you unix philosophy
   • [x] good community: e.g. this repo
   • [ ] community has agreed upon standards for common problems: code style, async handling, error propagation.
4. debugability: writing new stuff
   • [ ] great debugging tools: unless you're a dtrace wizard (and even then), node-inspector is about as good as it gets; but it's slow, and not very full-featured.
   • [ ] it's easy to get down to the metal if need-be: I love me the high-order languages, but sometimes I know better than v8 when to run GC; and other such problems.
   • [ ] stack traces are useful: async stack traces please.

Node-core may not be responsible for all of these. But it should probably, at the very least, have an opinion on all of them. IMHO

[ronkorving]

My biggest non-code related pain point is how Node 0.12 release was supposedly "imminent" for the past year or what, and here we are. I'm quite frustrated by the lack of communication from the project lead to the community. In fact, it feels like the community is pretty much being ignored right now, including those who contribute the most. Now I know I'm preaching to the choir here so I'll stop ranting and move on to what you're really asking.

My biggest pain point is asynchronous error handling. It's a real pain, and generators give a way to solve that. I would really like to see those in. I don't see a point in libuv.js as a standalone product, but who knows, if that does the trick. I'm not against asynchronous syntax in cases like EventEmitter on a server.

I'll go against the flow here a bit and give my 2 cents on Streams. I would like them to stay in core. It's so fundamental to thousands of modules, that taking it out and letting it "evolve" in different directions is asking for trouble. I think the fragmentation and incompatibilities could destroy what is currently a very successful ecosystem (npm). I'm already frustrated when I see coffeescript modules (which I refuse in my codebase) that are otherwise really useful. My frustration will reach new heights when I see a "tar" module that is only compatible with "substack-streams", but not "ogden-streams". For that I will need to download "ogden-tar". Please don't let that happen.

About me:

I'm VPoE at a company in Tokyo called Wizcorp, and we've been Node.js users since the days of 0.4, and have been using it on (game) projects with literally millions of users. We embrace open source, have open sourced a lot of work and contributed to a lot of others.

[joeybaker]

@ronkorving You convinced on streams. :+1: for them staying in core. Once you got into it, I realized: yes the streams implementation has been changed a few times, and yes it was tested in userland first, but they're such a core concept, that there really oughta be a "blessed" implementation.

[Qard]

I don't think they should be outright removed from core. Just that the existing API should be an abstraction of something lower-level that we can bypass, if we so choose.

Frankly, streams are super janky, but the current implementation is central to many, many modules. Removing the current API from core would break all of those, which would be unacceptable.

[bnoordhuis]

good docs: they're somewhere between barely-sufficient to a mess

Can you qualify that? Most people I talk to find the core docs adequate, at least as a reference guide. Or is your comment an observation about the larger node.js ecosystem?

[joeybaker]

@bnoordhuis Funny – most people I talk to say the core docs are not that great. I'm happy to provide an example.

---

Let's take the docs for http.createServer: http://nodejs.org/api/http.html#http_http_createserver_requestlistener

http.createServer([requestListener])#

Returns a new web server object.

The requestListener is a function which is automatically added to the 'request' event.

Some problems that I can quickly identify:

1. What is a "web server object"? Why do I want one?
2. What is the request event? Why do I care if a requestListener is added?
3. This is a pretty important method. Why no code example?
4. Bonus points: there should be a playground or quick video showing a server in action.

---

That's just one example. The docs are littered with problems like these.

[Qard]

Yeah, it's mostly just an API reference currently, and makes a lot of assumptions about what the reader already knows. I think the current docs should just be modified a bit to actually be pure API docs and then write something completely different to function as a beginners guide.

To compare to rust, what we have now is roughly equivalent to the 'reference' and 'API docs' sections merged into one thing, and we are completely lacking an official equivalent to the 'guide' section.

[rmg]

@Qard @Fishrock123 regarding generator performance, even if they were JIT-able, using try/catch disables optimization in v8, so you're still going to pay a penalty with generators+exceptions even if generators were optimized.

[domenic]

using try/catch disables optimization in v8, so you're still going to pay a penalty with generators+exceptions even if generators were optimized.

Not true anymore as of Turbofan.

[Qard]

What @domenic said.

And I'm not convinced that micro-benchmarking of generators is an adequate measure of performance, considering how much extra code is needed to achieve similar behaviour. The real-world performance will only be known when people start to properly experiment with it.

Anyway, I kind of regret bringing up the generator issue now, as I seem to have rather derailed the thread. Can we get back on-topic?

[Fishrock123]

regarding generator performance, even if they were JIT-able

That still doesn't answer if regenerator does or doesn't make JIT-able code :P

[zenparsing]

The JS landscape is changing fast - a year from now many aspects of the current API will seem very dated (e.g. CommonJS modules and event-based streams). The platform needs to evolve in order to adapt to that landscape.

While I think that a low-level core is a good thing, I also think that users need to have some fundamental higher level protocols defined by the platform so that components can interact effectively.

[mikeal]

The JS landscape is changing fast - a year from now many aspects of the current API will seem very dated (e.g. CommonJS modules and event-based streams)

I continue to be perplexed by comments like these. Has anyone actually spent time using the modules spec? It's objectively terrible from a user experience perspective, I've gotten spec authors to admit as much. As far as streams go, there's generator based streams, of which I've only seen early prototypes of, and there are the streams @domenic is working on at the W3C which are promise and DOM based, hardly a recipe for amazing ease of use.

I say this not to demean this work or say that it won't have an effect on Node, but all kinds of efforts have gotten traction after Node created its patterns, and in all of those efforts you see compatibility with Node's patterns implemented via modules and while Node's ecosystem grew tremendously because of the ease of use those other patterns enjoyed far less growth. My prediction is that the same thing will happen with modules, steams are a different story.

Whatever happens with streams at the W3C it's a given that we'll have to find a compatibility pattern between what exists today and what ends up in standards. That compatibility may be in the form a new simpler pattern that replaces wide usage in Node and in browsers. We have far more issues with stream compatibility right now than almost anything else. We have virtually no issues with modules, and are enjoying a pretty tremendous amount of adoption of those even in the browser.

The only thing core is failing at is shipping, which means we don't take new v8 versions early enough, which means we can't even try to create compatibility patterns without huge penalties (custom compiles, asking users to run with special flags). When you look at the adoption of co it's actually hugely impressive that it's done as well as it has given the current state of released generators in Node, and that's because it is by definition a method of compatibility between existing patterns in Node and generators.

[Fishrock123]

It's objectively terrible from a user experience perspective, I've gotten spec authors to admit as much.

I Agree. CommonJS makes so much more sense than the module spec..

[domenic]

@mikeal streams work is happening in the WHATWG, not W3C, and they are not DOM based. They are promises based precisely so you can do things like await ws.write().

[mikeal]

@domenic thanks for the correction. the politics between WHATWG vs W3C in particular escape me :)

[trevnorris]

@ronkorving

My biggest non-code related pain point is how Node 0.12 release was supposedly "imminent" for the past year [...]

Note: The following is not an excuse that v0.12 shouldn't have been cut 6+ months ago.

Needing to update V8 from 3.26 to 3.28 was required because of a couple critical bugs (e.g. V8 would segfault if using Promises in debug mode). But upgrading required us to implement our own debugger. @indutny has now taken care of that. I am working on the last critical change that must go in. Removing the user API for async listener (https://github.com/trevnorris/node/commits/remove-al-js). Finishing that is now my top priority (was up until 8am working on it) and should be finished by end of the week. I spoke w/ Joyent's CEO and let him know that once that patch was done all critical issue would be out of the way and v0.12 could be cut.

I'll go against the flow here a bit and give my 2 cents on Streams. I would like them to stay in core.

None of the current API would be removed (that would completely break backwards compatibility). There would just be an additional lower level API user's can hook into.

@Qard

And I'm not convinced that micro-benchmarking of generators is an adequate measure of performance, considering how much extra code is needed to achieve similar behaviour.

This isn't about benchmarks. The code itself will not allow generators to be passed to the optimizing compiler. So, generators could never be faster than callbacks.

[Qard]

Obviously a single generator would always be slower than a single callback. I'm not arguing against that.

For complicated things like streams though, the significant code reduction may actually make it perform better than you'd think. Just think of how many callbacks, among other things, are required to make the current streams implementation work. There's a lot of moving parts in there that wouldn't be needed.

I think the advantage of generators is really more about reducing cognitive burden than it is about performance though. If I really cared about such minor performance differences, I wouldn't be writing JS.

[Fishrock123]

I spoke w/ Joyent's CEO and let him know that once that patch was done all critical issue would be out of the way and v0.12 could be cut.

@trevnorris Cut from joyent/node or node-forward/node?

[rvagg]

@trevnorris word of warning on pushing a v0.12, as we're spinning up build machines for the new build infra we're running into lots of bugs beyond the standard platforms, even once you go from Ubuntu 14.04 to CentOS 6.5 there are bugs. Mostly minor I suspect and some will probably be problems with the tests themselves, but they'll need attention. I don't know if jenkins.nodejs.org is picking these up for you.

[zenparsing]

Has anyone actually spent time using the modules spec?

For sure!

It's objectively terrible from a user experience perspective

Well, that's quite clearly not objective (at least as far as the syntax goes), but regardless, once V8 ships modules it's not tenable to continue to hold out against them. Fortunately, I and many others have independently been working out how they can interoperate seamlessly with require. It's completely feasible. (If you're curious, see https://github.com/zenparsing/es6now/blob/master/docs/modules.md for one such example.)

On the streams front, you should also be aware of the experimentation going on with async generators. There are competing proposals in the works, but basically it will allow users to write generators which also include await expressions.

async function* asyncGenerator() {
    await something();
    yield somethingElse();
}

Exciting times for JS and Node!

[trevnorris]

@Fishrock123 From joyent/node. I plan on porting some of the commits from node-forward/node back before the cut.

@rvagg Are these issues that don't exist in v0.10? Are any of them critical (e.g. cause segfaults)? Otherwise I'm going to push to ship v0.12 and we'll prioritize bug fixes as they come in. If we need to release a v0.12.1 within a few weeks after that then so be it.

[calvinmetcalf]

@zenparsing not seamless

• calvinmetcalf/es6-translate#13
• esnext/es6-module-transpiler#69
• jrburke/requirejs#923
• cujojs/curl#236

[zenparsing]

[off-topic, briefly]

@calvinmetcalf It depends on what your interop goals are. ES6 modules aren't 100% fungible with CommonJS modules, so completely transparent interop is not really possible. But it doesn't need to be.

We should probably continue the discussion elsewhere though, if you're interested in my point-of-view.

[mscdex]

One of my major frustrations as someone who writes C++ addons for node is the lack of planning and roadmapping of backwards-incompatible changes for addons. I've yet to hear from someone of authority when/if we will see something like nan ship with node. There has only been a very brief mention of such a thing in a past nodejs.org blog post.

As an addon author, I would prefer to only have to convert my addons once and not convert them to nan now, only to have node core come up with its own, official, different abstraction later on. This is why I've been holding out on making my addons v0.11+ compatible (to the detriment of end users) ...

[bnoordhuis]

@mscdex It was discussed at yesterday's TC meeting. The tentative agreement we came to is to bless nan but not absorb it into core.

The current situation where nan is a dependency in your package.json is a good thing because it lets nan evolve at its own pace and makes it possible for add-ons to use older or newer versions of nan as they see fit.

There needs to be some coordination between nan and core but that should be no problem because both @rvagg and yours truly are closely involved with both projects.

Long story short, go upgrade your add-ons. :-)