Should design a new standard library?

[littledan]

These are some fundamental questions I've heard from folks when I discuss built-in modules with JavaScript developers. Have you heard these concerns, or any other concerns that aren't tracked in other issues? Do you agree with my suggested answers here? Is there more we should consider digging into here?

Q: Rather than make new libraries that are built-in, should we standardize existing libraries and put them in the platform? These meet real needs, have adoption in existing code (so speedups will come sooner) and are more ergonomic.

A: Let's figure this out on a case-by-case basis. Typical JavaScript libraries do things that we might not want to bake into the language, but we in the standards world have a lot to learn from them in terms of attractive ergonomics and practicality for direct use.

Q: Should the platform focus exclusively on low-level capabilities, rather than building a standard library? The ecosystem does best at higher-level things, being more agile and not having to worry about rollout across browsers.

A: This project has a longer timescale, so it is complementary to many ecosystem efforts. Built-in modules will be designed to be accurately polyfillable when possible, shrinking the time to become usable, and will have strong tests to ensure consistency across browsers. There's value in standardizing certain core components for reuse, for various motivations that are specific to each library being added.

Q: How will built-in modules avoid the pitfalls of previous standardization efforts, which result in poor ergonomics, not really solving developers' needs, etc.?

A: The standards processes have been improving, incorporating more developer feedback and prototyping, though we still have a way to go. Hopefully, being able to develop standard libraries in JavaScript will help get more people involved. Your ideas here are appreciated.

[littledan]

If there will be a standard library, make it like any other JS library and see if it can reliably compete with the rest of the ecosystem first.

I'm totally in favor of this last point. I don't think we have to wait until it's solidly winning 100% in the ecosystem, but some moderate amount of catching on would be good to see, or an understanding of why that didn't happen in a particular case or why something specific is more urgent.

[Qix-]

This seems like a slippery slope and ultimately a solution looking for a problem.

I couldn't agree more.

I don't see any benefit of this proposal other than shifting bloat around, not to mention potential performance hits due to JS<->C++ language boundaries.

If there will be a standard library, make it like any other JS library and see if it can reliably compete with the rest of the ecosystem first.

This.

I'm pretty shocked this hasn't been posted yet, but obligatory XKCD.

---

EDIT:

A quick note about left-pad.

I'm not going to sugarcoat it: it was a failure of NPM as well as all of the companies that treat NPM as a single point of failure. Our community champions fault tolerance but tends to show very little of it when it comes to our development process.

When I worked at Uber, we were immune to NPM outages or silly things like packages being unpublished since we used a CouchDB mirror, which is painfully easy to set up and maintain I might add.

When I develop native code, I usually mirror repositories to my own account for similar purposes if I'm using submodules, or I'll vendor the code directly into my codebase. This has little to no overhead aside from maybe a few more seconds during a clone, which happens once every month at most.

Further, the 'problem' with unpublishes has been patched and is thus a moot point. I don't see how it has any bearing here.

---

Another note:

Changing the language to solve the shortcomings of the package manager is a very quick way to ruin the language.

Pip, easy_install, anaconda, virtualbox, etc. - they all provide something the other doesn't, and working with all of them in a project is a headache, sure.

But that doesn't mean Python should be changed.

I think the only viable argument here is that there are many small modules that exist to patch small pieces of functionality in the standard types. However, saying we need a full-blown standard library to fix the problems of node_modules, unpublishes, et al is a ridiculous jump to conclusions, in my opinion.

---

In response to the points in https://github.com/tc39/proposal-javascript-standard-library/issues/19#issuecomment-454402340:

They're a standard spec, thus widely implemented everywhere (cross-platform), so they can be relied upon.

That's literally the point of Javascript. There's nothing new here.

They can be implemented to be faster/with less memory usage.

[citation needed]

This is speculation at best.

They do not need to be bundled with the user's code, thus reducing file sizes (a big deal with JavaScript).

While not directly in the node_modules, they're still now part of the runtime, which is just shifting bloat elsewhere.

The net effect of this is that the code is hidden elsewhere, and instead of having a single implementation that you have inspected and know works with your project, you now have multiple platforms with potentially differing implementations of the functionality.

This is a net-negative if you ask me.

They significantly ease prototyping (since you don't need to fiddle with npm, installing dependencies, bundling, etc.)

If you're "fiddling" with npm, you're probably using it wrong or are setting up your machine for the first time.

If you're prototyping, you're probably not bundling.

Installing dependencies is a fact of life, for any language.

They significantly ease tweaking, debugging, and experimentation, since the features are available right in the Node/browser's dev console.

This doesn't make sense. Aren't dependencies also in those places? You're arguing that we should add a massive development overhead to JS engine vendors for the sake of saving a few seconds to avoid a call to require().

They make writing small shell scripts (or quick one-off scripts) dramatically easier.

How?

They significantly ease teaching for beginners (since beginners can just jump right in and start writing code, without dealing with all the complexity of libraries and package managers right away).

Ease of use for the general public should be the focus of any language, not focusing on how beginners approach a topic.

Beginners are going to have to understand dependencies, importing, etc. at some point in their journey, what good is it to shield them from it? Also, how does this have anything to do with the formal specification of a major programming language?

[PJB3005]

not to mention potential performance hits due to JS<->C++ language boundaries.

Why would the standard library HAVE to be implemented in C++ and by extension incur overhead? Surely browser authors are free to implement it in the most optimized way possible. If it were slower for the browser to implement in C++ then they could just... implement it in JS, and it'd be transparent to you anyways.

Changing the language to solve the shortcomings of the package manager is a very quick way to ruin the language.

This problem isn't due to a shortcoming of the package manager though, it's due to a shortcoming of the language.

saying we need a full-blown standard library to fix the problems of node_modules, unpublishes, et al is a ridiculous jump to conclusions, in my opinion.

While yes those are some of the problems it aims to solve, it aims to solve a lot more too.

While not literally in the node_modules, they're still now part of the runtime, which is just shifting bloat elsewhere.

Except now the bloat doesn't have to be sent over the wire constantly and can be much more efficiently loaded by the JS runtime. I don't know what kind of amazing tricks browsers employ nowadays, but I can't imagine it being slower than polyfills in pure JS.

The net effect of this is that the code is hidden elsewhere, and instead of having a single implementation that you have inspected and know works with your project, you now have multiple platforms with potentially differing implementations of the functionality.

This is true, but I think this problem can be mitigated significantly. For example a standard test suite could be made that browsers have to comply with. Also this is already the exact problem the web faces, and I don't think "we shouldn't add features to browsers because it might have buggy implementations" is a common argument.

Installing dependencies is a fact of life, for any language.

Perhaps, but in no way to the extent with JS. In something like .NET you can get very far with just the standard library. Maybe a large framework depending on what you're building.

This doesn't make sense. Aren't dependencies also in those places?

I think what was meant is the case where you want to use some quick code in your browser's dev console to try something out, but to write that you need a dependency that you didn't bundle. Which the standard library would of course avoid.

And finally for the relevant XKCD: The difference here is that that doesn't apply when the new standard will be the future universal solution. If people used this argument against USB then maybe we'd still have charger spaghetti.

[Qix-]

This problem isn't due to a shortcoming of the package manager though, it's due to a shortcoming of the language.

Half the arguments here refer to npm problems.

While yes those are some of the problems it aims to solve, it aims to solve a lot more too.

Care to elaborate?

Except now the bloat doesn't have to be sent over the wire constantly and can be much more efficiently loaded by the JS runtime. I don't know what kind of amazing tricks browsers employ nowadays, but I can't imagine it being slower than polyfills in pure JS.

Over the wire, perhaps. That's the only valid point, as you're contradicting your prior statement - if it's included as a javascript library, it means that each isolate is going to have to load in the standard libs anyway.

For example a standard test suite could be made that browsers have to comply with

Ah yes, I love waiting for Microsoft to implement features incorrectly 2 years after they're mandated.

What will ultimately happen is that this standard library will be decree and we'll spend many extra hours writing polyfills for it because browsers/environments/runtimes don't get them all, or don't get them all correct.

we shouldn't add features to browsers because it might have buggy implementations

"Might" is too loose a term. There will be buggy environments, overhead, and periods of waiting, especially since I would assume this "standard library" would be organic.

This is no different than requesting things be added to the built-in prototypes.

There are decades of evidence this would not be a smooth rollout process.

In something like .NET you can get very far with just the standard library.

Which is why projects like Mono have a really hard time supporting all of the features. Same with OpenJDK at first. It took decades for them to get it right.

If people used this argument against USB then maybe we'd still have charger spaghetti.

Software and hardware standardization problems are quite different, firstly.

Secondly, I think you're quite wrong here. Hardware standards are competing. They allow developers to make appropriate tradoffs for various manufacturing benefits. Software has a much different landscape when it comes to standards, as we tend to favor compatibility since our "manufacturing" overhead is virtually zero outside the realm of infrastructure and labor, which aren't being address by this proposal whatsoever.

That's an apples-to-oranges comparison.

[Nimelrian]

Half the arguments here refer to npm problems.

npm problems which stem from a lacking standard library, which motivated people to write single-digit line packages. This isn't a "ridiculous jump to conclusions", it's simply backtracing from the current mess that is the JS package ecosystem.

I really don't see the huge problem people have with adding a mechanism (Note that this proposal is about the mechanism, not about what would be added to the libraries, which can then be loaded by said mechanism) to import standardized functions / classes / whatever.

[evanplaice]

npm problems which stem from a lacking standard library, which motivated people to write single-digit line packages

Strawman. The need to keep libs small (ie pre tree shaking + ESM) combined with laziness was the cause of a single line lib breaking production code.

We're apprpaching a new era of JS lib development where designing larger aggregate libs with mostly dead code will no longer have a negative affect on performance.

Once native ESM is made available. It will be perfectly feasible to create a standard lib available via NPM.

Emphasis on - NOT IN CORE.

That way:

• universal compatibility is guaranteed
• upgradeability is simple
• contributing is easy
• platform/spec bloat is minimized
• breaking change can be made on major versions
• the progress/versioning doesn't have to be tied to Nodes release cycle
• lib backward compatibility can be maintained independently of platform compatibility

If the standard lib becomes so damn essential to the JS ecosystem that it would be a detriment to the ecosystem to NOT include it - then, and only then - should it be included in CORE.

[kaizhu256]

tree-shaking is a marketing-gimmick / joke. it has and will never work with a dynamic-language like javascript. I've manually tree-shaked various libraries using code-coverage analysis (swagger, nedb, etc...), and concluded a non-human could not possibly figure out the many circular and dynamic code-references common integration-level javascript.

[Qix-]

tree-shaking is a marketing-gimmick / joke.

Speaking academically, yes and no. "Tree shaking" is a marketing term for dead code removal. There are situations where you can assert that code is dead in Javascript with guarantees about side effects. But those cases are rare.

Unfortunately, Javascript has way too many ways to introduce subtle or downright hidden side effects that can affect dead code removal provability.

I do agree, however, with the sentiment that tree shaking doesn't really have a spot in this conversation - I see it as beside the point, much as I think anything related to npm is also beside the point.

Also, I don't see how "bloated node_modules omg" is a problem solvable by the language. Native applications have the same problem when you distribute source. The size comparison comes down to bundling - in native applications "bundling" is compilation. You compile to an executable, which in many cases is leagues smaller than your sources.

With Javascript, bundling is comparable to the size of the source code, sans readmes/licenses and not taking into account compression. This is because javascript doesn't have a binary format (shout out to https://github.com/tc39/proposal-binary-ast).

This is also why WebAssembly was attractive to many people.

[evanplaice]

Speaking practically, not academically. Forget 'tree shaking' there is a greater issue here.

Specifying that the standard library is built into the platform implies that the spec will need to be universally applied to most/every runtime that supports JS. Including node, browsers, embedded, and any devices that runs JS. Also note, Javascript's application/usage will likely increase to cover even more platforms over time.

This compounds the maintenance overhead on a language ecosystem that is already struggling to keep up with existing/emerging standards.

Also, I don't see how "bloated node_modules omg" is a problem solvable by the language.

I don't see how shoving what is essentially a global library into the runtime solves the issue either.

This top-down, 'maintainers know what the users need' approach has been done already (ref .NET/JRE/C++). From existing examples; standard libs tend toward massive, bloated, monolithic catch-all collections of utilities that become a 'necessary evil' of working in the language.

The advantage of doing it in JS using the existing tooling/ecosystem, is that it will be easy to exclude and/or dispose of the std lib if the experiment turns out to be a failure.

I'm not saying it should never be included in the runtime. Only that it's sensible to prove that a std library is necessary to begin with, prior to specifying it as a long-term maintenance commitment for all platform creators/maintainers.

With Javascript, bundling is comparable to the size of the source code, sans readmes/licenses and not taking into account compression. This is because javascript doesn't have a binary format (shout out to https://github.com/tc39/proposal-binary-ast).

Yeah, this would be really awesome. If a warmed cache containing the (already downloaded->parsed) AST representation of libs existed, it might actually be feasible to go back to the 'old way' of providing common library LTS releases via CDN.

[Qix-]

@evanplaice I think you misunderstand me.

This compounds the maintenance overhead on a language ecosystem that is already struggling to keep up with existing/emerging standards.

I agree.

I don't see how shoving what is essentially a global library into the runtime solves the issue either.

I agree, it doesn't. Neither the package management nor the "more is better" arguments here make any sense to me.

The advantage of doing it in JS using the existing tooling/ecosystem, is that it will be easy to exclude and/or dispose of the std lib if the experiment turns out to be a failure.

I agree.

---

This proposal is a solution to an X-Y problem. I don't see it any other way.

[evanplaice]

@Qix- Right, my bad. I interp'd it as you were in favor of the 'JS libs suck, put it in the runtime' approach. I'll quit with the broken record then, see what other peeps have to say.

[aditya1944]

Atleast normal queue and priority queue should be implemented in the core. These are common problems and developer don't have to spend time on implementing and using these.