Should design a new standard library?

littledan commented 5 years ago

These are some fundamental questions I've heard from folks when I discuss built-in modules with JavaScript developers. Have you heard these concerns, or any other concerns that aren't tracked in other issues? Do you agree with my suggested answers here? Is there more we should consider digging into here?

Q: Rather than make new libraries that are built-in, should we standardize existing libraries and put them in the platform? These meet real needs, have adoption in existing code (so speedups will come sooner) and are more ergonomic.

A: Let's figure this out on a case-by-case basis. Typical JavaScript libraries do things that we might not want to bake into the language, but we in the standards world have a lot to learn from them in terms of attractive ergonomics and practicality for direct use.

Q: Should the platform focus exclusively on low-level capabilities, rather than building a standard library? The ecosystem does best at higher-level things, being more agile and not having to worry about rollout across browsers.

A: This project has a longer timescale, so it is complementary to many ecosystem efforts. Built-in modules will be designed to be accurately polyfillable when possible, shrinking the time to become usable, and will have strong tests to ensure consistency across browsers. There's value in standardizing certain core components for reuse, for various motivations that are specific to each library being added.

Q: How will built-in modules avoid the pitfalls of previous standardization efforts, which result in poor ergonomics, not really solving developers' needs, etc.?

A: The standards processes have been improving, incorporating more developer feedback and prototyping, though we still have a way to go. Hopefully, being able to develop standard libraries in JavaScript will help get more people involved. Your ideas here are appreciated.

JonForest commented 5 years ago

Personally I'm very keen on extending the standard library. I expect a large part of the node_modules bloat I think comes from little modules patching common gaps. A common, approved, safe set of functions helps protect us from the leftpad fiasco of a couple of years ago.

ljharb commented 5 years ago

@JonForest No, it doesn't; the issue with left-pad had zero to do with a hole in the standard library (one that's now filled by padStart), and was entirely to do with an author's ability to unpublish at will (which is now no longer possible after a few days, and thus no longer a problem in the future)

littledan commented 5 years ago

Let's work together to solve the parts of the problems where they can be solved. We can add new standard library features with work in this repository, and give feedback on npm in https://npm.community/ .

evanplaice commented 5 years ago

The best part about JS is that there is no standard lib.

Take jquery as an example; a decade ago the entire JS community probably would've agreed that JS and JQuery are practically synonomous, I even developed a relatively popular lib that became popular in large part because it masqueraded as a jquery extension.

Nowadays, with better module support in ES6. The ability to tree shake out unused code from the output. The upcoming functional/pipeline syntax extensions. Etc... If jquery were rewritten today it would be designed fundamentally different and much better. IIFE's and prototype chaining used to be ubiquitous whereas now they're fundamentally bad practises.

Take lodash as an example. The new pipeline syntax for functional-like async processing is cleaner, more intuitive than the current rendition and will likely spark a new generation of utility libraries like lodash that will be more ES6/Tree Shaking friendly.

At best, a standard lib will inevitably become baggage over time as newer and better solutions are developed/discovered.

yogevyuval commented 5 years ago

@evanplaice While I agree with what you are saying, it seems that the state of JS standard library is a bit behind other full featured languages like Python, Java and Rust. Mostly because of really surprising and basic features that you expect a language so powerful to have, and end up using lodash instead or writing a utils.js file to your project.

I think a lot of people agree that JS is really great especially with ES6, but it can be even better with a bit more work to the std, while not making it a bloated baggage.

OlivierJM commented 5 years ago

For the case of lodash and possibly date-fns. As mentioned they might be improved sooner or later as more better solutions are developed like the pipe operator and others. Why isn’t the concept adopted rather than the current existing code of these libraries.

it would really make it easy to have such as part of the core language

martinheidegger commented 5 years ago

A: Let's figure this out on a case-by-case basis....

This answer is avoiding the conversation. Do you want to the community to open a "case" for every library that should make it's way upstream? Then this would answer better:

"We'd be happy to consider elevating existing libraries. Please add an issue in this repo so we can discuss whether that library is good to consider for standards."

If not, maybe something like this might suit you better:

"We are looking into the solutions brought up by the ecosystem but ultimately attempt to build with the community something new that fulfills our quality standards."

A: This project has a longer times...

This is not answering the question: "Should the platform focus exclusively on low-level capabilities?". From what I have read this may rather be:

"The standard library is focusing on low-level capabilities! The issue is that by adding more and more features to the JavaScript core we risk startup-performance issues of JavaScript engines. We want to provide a way to developers to choose which parts of our work they wish to use."

littledan commented 5 years ago

Those answers look great! Maybe you would be a better person than me to make the PR to put an FAQ in this repo.

martinheidegger commented 5 years ago

I can help reviewing any PR that you open 😉

calebdwilliams commented 5 years ago

I want to throw my hat in with @evanplaice. This seems like a slippery slope and ultimately a solution looking for a problem. With the new standards processes that are in place, I don't ultimately see what this proposal would solve well (and the things it would solve introduce myriad complexities).

While I will agree with the fact that we need utilities like map and len and time utilities, I don't think importing them from a standard library that is somehow separate from JS core really does anything for developers that the current process doesn't allow. map and len could be added to Reflect, a new Time object could be created or time utilities added to Intl.DateTimeFormat.

The one real benefit might be versioning, but any utility that needs to be versioned might not belong in a standard library any way.

I believe implementing this proposal would result in a fractured experience for developers.

martinheidegger commented 5 years ago

@calebdwilliams I am trying to understand your comment and put it into question form. Is it correct that your question would be:

Q: What advantages does the standard library have over simply adding functionality through the new standards process?

calebdwilliams commented 5 years ago

@martinheidegger More or less.

serapath commented 5 years ago

PLEASE STOP IT!!!!

Any developer who moves into TC39 or any other standards body is basically avoiding competition and wants an easy way to PUSH their ideas to EVERYONE in the entire world. This is unbelievably rude! If you have an awesome idea, make a module, market it to other devs like everyone else does and if they like it they will adopt it.
Userland exists so that the core can stay lean. Different people have different needs. The core needs to solve what cant be solved in userland. The only reason to add to the core is to enable useful features that can't be achieved without. Already today that rule is violated way too often and TC39 is abused.
Maybe you can Lobby somewhere else to add some kind of cross domain module cache to javascript environments, so that a downloaded "module" used in one context can be re-used in another and doesn't need to be downloaded again.
If such a mechanism exists, certain feature - whether new Map or new Set or new Array or in browsers window[XXX] could be a blocking lookup to fetch immutable modules dynamically from a standards body like w3.org or otherwise from configured vendors like firefox/chrome/safari/etc... or alternative endpoints (maybe npm?) AND then cache them and make them available to other programs too (e.g. a different browser tab loading a different domain or different node process should be able to re-use it)

Would't that solve all the problems?

Nimelrian commented 5 years ago

PLEASE STOP IT!!!!

Any developer who moves into TC39 or any other standards body is basically avoiding competition and wants an easy way to PUSH their ideas to EVERYONE in the entire world. This is unbelievably rude! If you have an awesome idea, make a module, market it to other devs like everyone else does and if they like it they will adopt it.

The "make a module for every single idea" approach has led us to the horrible NPM ecosystem, which has and is causing issues like the left-pad fiasco, a bloated node_modules folder and incredibly deep dependency trees.

Userland exists so that the core can stay lean. Different people have different needs. The core needs to solve what cant be solved in userland. The only reason to add to the core is to enable useful features that can't be achieved without. Already today that rule is violated way too often and TC39 is abused.

Userland exists to achieve certain use cases using the means provided by the language (including its standard library). The goal of this proposal isn't to implement specific solutions for specific problems, but common solutions to common problems.

The standard library is also separate from the language. Language evolution e.g. by adding syntactic sugar isn't a bad thing, but often enhances maintainability and usability of the language itself.

Maybe you can Lobby somewhere else to add some kind of cross domain module cache to javascript environments, so that a downloaded "module" used in one context can be re-used in another and doesn't need to be downloaded again.

If you're using yarn, you can already make use of that. This is a topic which is far outside the scope of the language committee.

If such a mechanism exists, certain feature - whether new Map or new Set or new Array or in browsers window[XXX] could be a blocking lookup to fetch immutable modules dynamically from a standards body like w3.org or otherwise from configured vendors like firefox/chrome/safari/etc... or alternative endpoints (maybe npm?) AND then cache them and make them available to other programs too (e.g. a different browser tab loading a different domain or different node process should be able to re-use it)

See my answer above.

Would't that solve all the problems?

No, it would create a multitude of new ones.

ljharb commented 5 years ago

@Nimelrian since everyone seems to misunderstand the left-pad incident, please see https://github.com/tc39/proposal-javascript-standard-library/issues/19#issuecomment-447970283.

Note that npm has the same machine-level cache as yarn for down loaded archives.

Nimelrian commented 5 years ago

@ljharb No, I understand it exactly the way you described it.

The chain of events which lead to the incident has a simple root: A (at that point in time) lacking standard library. The horrible (non-existent) security model of NPM is obviously a different issue, but it served mainly as an enabler, not as the root.

The small standard lib is exactly what led to the huge amount of packages and the huge dependency trees we see nowadays. Missing functionality is "fixed" by writing a package for it and people import it, because they don't want to write "things which should be standard" themselves.

Just compare dependency trees of Java or C++ projects, which have huge standard libs, with those of JS projects. Most of the dependency bloat in JS comes from packages which implement small, common solutions to common problems, which is exactly the thing a standard lib is meant to do.

martinheidegger commented 5 years ago

@serapath Attempting to put your concerns into question form for a Q&A. I don't want to attempt giving answers!

Any developer who moves into TC39 or any other standards body is basically avoiding competition and wants an easy way to PUSH their ideas to EVERYONE in the e....

"A lot of effort has been put by individuals to promote new ideas. Many of those innovators get nothing out of their effort but the fame of their library. The TC39 would gain the power to render an existing effort mute and thereby ruining a life's work. What do you intend to do to prevent private agendas resulting into half-baked and/or derivative solutions that might do more harm to the ecosystem than good?"

Userland exists so th... way too often and TC39 is abused.

"The bigger a core library grows, the harder it becomes to maintain, secure and update. Why do you think it is a good idea to increase that scope?"

(I felt this was slightly redundant to the former Q, reduced to non-redundant question)

Maybe you can Lobby somewhere else to add some kind of cross domain module cache to javascript environments, so that a downloaded "module" used in one context can be re-used in another and doesn't need to be downloaded again.

"Why is it not possible to distributed a good standard library separately from the vm? What drives you from finding solutions that utilize a flexible distribution mode in favor of stiff solution?"

If such a mechanism e...

"A good flexible and secure loading mechanism could allow the greater eco-system to piggy-back on your efforts, wouldn't that be a worthy goal?"

(the last one was really hard to put into question form)

mcollina commented 5 years ago

I think JS not having a standard library so far has been one of the greatest things to make the broad JS community so innovative. I think extreme care needs to be taken to add things to the standard library, mostly working to standardize/add the most common patterns that are available in the ecosystem.

From my purely-Node.js point of view, I think the goal of the standard should be to enable the evolution of the language/platform without polluting the global namespace. I would love to see a clear line between the JS standards (TC39) and the Web standards (WHATWG) and all other globals that are currently added by runtimes (e.g. Buffer in Node.js). Not having this distinction makes it hard to develop isomorphic applications.

TheLarkInn commented 5 years ago

I agree heavily and on behalf of webpack we would be strongly against a standard library.

The modern module and package management systems we have today would ensure Stdlibs are always outdated and obsolete by their alternatives. Keeping the core small enables a rich ecosystem.

Akamaozu commented 5 years ago

This seems like a heavy-weight solution.

Instead of a new stdlib which is almost guaranteed to fracture the ecosystem even more, why not have a list of recommended modules on npm for different usecases?

E.g.

Date/Time - moment HTTP request - request HTTP server - express

This list can be updated annually, and the old list permanently archived.

Think awards show, but for packages.

Pauan commented 5 years ago

@Akamaozu There already is:

https://github.com/sorrycc/awesome-javascript#readme https://github.com/sindresorhus/awesome-nodejs#readme

In fact there's even a list of lists:

https://github.com/sindresorhus/awesome#readme

Akamaozu commented 5 years ago

Are any of these run or publicly endorsed and promoted by TC39?

Pauan commented 5 years ago

@Akamaozu No, and to be honest they shouldn't be: the TC39 committee has its hands full.

They are already struggling just to keep their own proposals up to date.

They should focus on improvements to JavaScript, not ecosystem curation.

Having it be a community-run wiki ensures that the quality stays high and that it remains up to date.

Having it be community-run means that it's very easy for ordinary people to contribute, which is how it should be.

If you simply meant that these lists should be more widely promoted and known within the JavaScript community, then yes I agree. However, I don't think that requires TC39 promotion. Word of mouth is plenty enough.

Akamaozu commented 5 years ago

So basically let them keep up the busy work of creating a new stdlib, but ...

if somehow these lists were popular enough they wouldn't need to
they don't need to publicly endorse or promote one because word of mouth is enough

Let me ask this:

What is a stdlib if not a list of recommended util libs rolled into core?

Pauan commented 5 years ago

So basically let them keep up the busy work of creating a new stdlib

I never mentioned anything about whether TC39 should create a stdlib or not, I merely replied to your suggestion about creating a list of curated npm modules.

What is a stdlib if not a list of recommended util libs rolled into core?

There are many differences, especially with regards to versioning and out-of-date libraries. But I'm not trying to get into an argument about that.

Nimelrian commented 5 years ago

@mcollina

I would love to see a clear line between the JS standards (TC39) and the Web standards (WHATWG) and all other globals that are currently added by runtimes (e.g. Buffer in Node.js). Not having this distinction makes it hard to develop isomorphic applications.

The stdlib proposed here would be part of the JS standards you mention and would contain nothing related to the web API or other runtimes.

@TheLarkInn

I agree heavily and on behalf of webpack we would be strongly against a standard library.

The modern module and package management systems we have today would ensure Stdlibs are always outdated and obsolete by their alternatives. Keeping the core small enables a rich ecosystem.

The stdlib would/should only implement certain core functionalities (utility functions, containers, etc.) which can be tested easily. The purpose of the stdlib wouldn't be to replace entire libraries/frameworks, but to supply small solutions to small problems.

If a third-party solution offers better features or fixes a bug in the stdlib implementation, people can use that and the stdlib can be updated in time.

I think many people here misunderstand the goal and purpose of a standard library. It wouldn't replace entire frameworks. It would only supply well-tested, single implementations for common problems in our everyday use of the language.

calebdwilliams commented 5 years ago

@Nimelrian What, then, is the benefit of adding these sorts of utilities to a "standard library" rather than adding them to the core?

From my vantage point, those benefits seem to be almost nothing especially when weighed against the cognitive cost of introducing a new paradigm into the JavaScript ecosystem. Yes, this feature might work for other languages, but in its current conception, it doesn't seem to fit with what JavaScript is and has been.

Nimelrian commented 5 years ago

They will be built "into the core" as in "They'll be part of the language specification". That doesn't mean they'll be available on the global scope, but instead have to be imported explicitly.

To add:

And I'd be glad about this. Having to pull in (sometimes multiple versions) of utility libraries like lodash hurts page performance and puts people at the mercy of the maintainers of these libraries (e,g, lodash is gonna remove the omit function, which is widely used in FP or the React world, in version 5).

Another advantage would be that implementations can optimize the standard library functions using native code. That's impossible for userspace libraries.

calebdwilliams commented 5 years ago

@Nimelrian, apologies for not being clearer (the fact that there was confusion I think proves my point though). What I meant to ask is what benefit there would be to adding new features to a "standard library" as opposed to the global scope?

That's currently not how JavaScript works, but that's beside the point.

In response to @TheLarkInn, you said

The stdlib would/should only implement certain core functionalities (utility functions, containers, etc.) which can be tested easily. The purpose of the stdlib wouldn't be to replace entire libraries/frameworks, but to supply small solutions to small problems.

Which is exactly what the current standards process is designed to do. Passing these new features into a otherwise hidden "standard library" just creates a bifurcated developer experience.

Having to pull in (sometimes multiple versions) of utility libraries like lodash hurts page performance and puts people at the mercy of the maintainers of these libraries (e,g, lodash is gonna remove the omit function, which is widely used in FP or the React world, in version 5).

Two points here:

dependency maintenance is part of the job and even if TC39 adds a versioned standard library, it's plausible that this same issue comes up, but with built-in library code (and if something needs to be versioned, it doesn't belong in the core language)
when you distill this argument, it's ultimately begging the question. How does attaching these functionalities create a better developer experience than adding them to the global scope? If it doesn't or if it is merely a matter of a handful of developers' preferences, I think the correct position would be to defer to how the language currently works/operates because the cognitive overhead is not worth the tradeoff.

tbremer commented 5 years ago

Having read the proposal and this thread, I am left wondering a few things…

What is the benefit of implementing a standard library above adapting and increasing the prototypes of already existing types?
- That is if map is something Set and Map should implement how does the standard library do that better than the prototype chain?
How can this be implemented in a way that does not have impactful breaking changes or unnecessary misdirection for new users of the language?
- Older code that is unmaintained must still be able to execute w/o error.
- Introducing (something like) Date (referencing the README) in a standard library that is somehow, at least in appearance, different than the (window/global) Date constructor adds a level of indirection that is a degradation of the current experience of writing JavaScript.
If this proposal makes it to Stage 4, where do adaptations of the standard library go?
- If we are adding something new to the std lib is that a proposal for std lib or proposal that references std lib?
- If we are adding a new primitive or global container is that a proposal for the JS language or the JS standard library?

Nimelrian commented 5 years ago

What I meant to ask is what benefit there would be to adding new features to a "standard library" as opposed to the global scope?

Adding new things to the global scope always bears the danger of breaking existing code which may add symbols to the global scope as well. The global scope should be kept as clean as possible. Implementing the stdlib as something you can import things from solves this problem.

What is the benefit of implementing a standard library above adapting and increasing the prototypes of already existing types?

Not adding things to prototypes but instead importing them as free functions leverages things like higher order functions, reusability, and composability.

How can this be implemented in a way that does not have impactful breaking changes or unnecessary misdirection for new users of the language?

By not adding stuff to the global scope / prototypes, but instead pulling it in explicitly as free functions / new constructors with their own prototypes

calebdwilliams commented 5 years ago

@Nimelrian, so too would adding those objects and functions to an object on the global scope (even if that object were named stdlib). The idea that you should need to import something built into the creates an entirely new paradigm in JavaScript that is unnecessary and will add a significant amount of cognitive overhead that will negatively impact the cohesiveness of the language.

Pauan commented 5 years ago

@calebdwilliams ES6 modules are a standard part of JavaScript, they are implemented in browsers, currently being implemented in Node, and already widely used throughout the npm ecosystem (thanks to things like Webpack). They are not some bizarre esoteric new feature.

So yes, it is technically true that using ES6 modules for browser built-ins is new, but it's certainly not unusual, and Node has been using modules for built-ins for decades. And other languages have used modules for built-ins for decades. It's not a new idea.

And since user code uses ES6 modules, having the browser built-ins be ES6 modules actually increases consistency and cohesiveness, rather than having them be these weird global things which behave differently from user code.

Lodin commented 5 years ago

@calebdwilliams, why do you think so? Any update itself is a cognitive overhead, but we have a ES2015 and it's great.

We all know why god objects are wrong. Especially mutable god objects. And JavaScript used to have it by design from the very start. However, if a lot of features built around this architectural flaw doesn't make it less flaw. It just means we cannot undo it in the next release. But we still can incrementally fix it, and introducing the standard library is a great step.

And I don't really think it is a big overhead. We import things. We do it all the time. So if we can import from non-existing file, what does it change? It still keeps the paradigm introduced by ES2015 modules.

And I'm very excited to have a possibility to have a standard library onboard. I really tired to choose and install simple things that all other languages includes from the box. And I really don't see any disadvantages of it.

Poor design? It can be handled by discussions and ideas. Moreover, JavaScript is on the stage this problem is solved so many times it's almost impossible to make it really poor. Lodash, ramda, underscore, tons of them. Just take and use. Other claims to design a just a matter of taste. How much map implementation did we have before Array.prototype.map? Where have they gone now? They just became unnecessary even if someone don't like that native map don't have some abilities lodash's map has.

These functions also will be faster than analogues. There will be no additional size forces us to write a lot of webpack plugins just to keep lodash from blowing up our packages. And that's even more great.

Also, it's quite polyfillable. Just install the package with the name of standard library, and you're done.

calebdwilliams commented 5 years ago

@Pauan You do realize that the guy who basically runs Webpack is on the record earlier in this thread against this idea, right?

I will concede the Node argument is the strongest example of why this should be implemented, but if we examine the places where Node added core functionality as a module, it was in places that only Node could feasibly use or needed Node-specific implementations.

As for @Lodin, modules are fine, they work great for ES and I literally use them every day. The success of Array.prototype.map is a great example of how the current standards process is working without the need for a "standard library." I'll reiterate the point, if something needs or might need to be versioned, it doesn't belong in the spec. Period. If we want a set of utilities like Lodash or any of the others you mentioned, they can be built in with the current global object/prototype methods that are used.

My criticism is about language cohesiveness and adding this feature would make JavaScript less cohesive.

Pauan commented 5 years ago

@calebdwilliams Of course, I am well aware. His objections were about the stdlib becoming outdated, which is a very real concern (e.g. see the Python stdlib, which has been outdated). That has nothing at all to do with ES6 modules vs globals (globals would also become outdated).

To be clear, I am generally against a JS stdlib. I think certain specific functionality should be added (such as more Array.prototype methods, some generator functions, maybe a few helper functions), but a rich stdlib should be avoided. The ecosystem does a better job with that.

But if functionality is added, it should be done through ES6 modules. That is my point.

calebdwilliams commented 5 years ago

@Pauan I completely agree, but this is kind of an anchoring argument. If we can agree that things should be on a global, then what's the use of a stdlib object? Just put it on the prototype or on the Reflect object.

Pauan commented 5 years ago

@calebdwilliams As shown in the README, the stdlib is just using ES6 modules, no globals, no stdlib object, no Reflect methods:

import { Date } from "std:Date";
import { len, map } from "std:builtins";

This is what I am in support of. The only exception is for adding methods to existing objects (such as Array.prototype).

calebdwilliams commented 5 years ago

@Pauan and that's what I'm arguing is unnecessary. len and map make more sense to me as part of Reflect, date utils make more sense as static Date methods or part of Intl. I'm not at all against standardizing these utilities, quite the opposite in fact.

What I am opposed to is adding them to the junk drawer that a "standard library" will inevitably become. If it does get added, though, I agree using modules and the layered API proposal linked in the README is the right way to do it, but I don't believe that "If" should actually come about.

EDIT: At this point, I think my position is clear. I'll leave it up to the standards body and community to continue this discussion (and only jump in if something else is added). Either way, I do sincerely appreciate the thought and work that goes into making JavaScript such a successful language. Thanks all for the insightful conversations and happy holidays!

ljharb commented 5 years ago

(Please note that Reflect is solely for being a 1:1 correspondence with Proxy traps; so nothing can ever be added there unless it does)

yacinehmito commented 5 years ago

I've been lurking in this issue; apologies for chiming in.

So far there has been talks of "date utilities", "map", "len", and in general quoting @Nimelrian :

The stdlib would/should only implement certain core functionalities (utility functions, containers, etc.) which can be tested easily. The purpose of the stdlib wouldn't be to replace entire libraries/frameworks, but to supply small solutions to small problems.

It would be great if people would define what they mean by standard library first if we want that discussion to ever move on. Wouldn't the opinions of everybody be heavily dependent on that?

I least mine would be. @littledan what do you have in mind when you say standard library?

littledan commented 5 years ago

Well, we already have a standard library in JavaScript, it's just very small. Things like RegExp and Date. IMO we should do whatever is most useful; there are lots of different goals we can consider. Small solutions to small problems sounds like a great way to start.

noncombatant commented 5 years ago

I'd love to see more things like RegExp and Date, and I'd love to see additional useful methods added to existing object prototypes, and I'd love to see the new things added into their own namespace(s) instead of added to the global namespace. To me, all those things fall under the umbrella of "standard library".

The heavy dependencies that come with Node modules really hurt safety and performance, and looking around it seems to me that a lot of those modules are filling gaps that other languages have filled in their standard libraries.

Certainly, there will be things that are not yet ready to be elevated to standard library status; Node modules and other things can continue to fill that niche, just as (e.g.) Boost has for C++. When things mature, they can be elevated (as many things from the community have been in C++ and Go). That's how other language communities are doing it, and it has proven to work, and it saves a lot of developer (and user!) time.

AtibaBryan commented 5 years ago

I wonder how hard it would be to get an official js standard library that's published on npm so you can have a lean core for low memory environments but have an easy way to add a library that's blessed/secured by a trustworthy org.

mw3i commented 5 years ago

I wonder how hard it would be to get an official js standard library that's published on npm so you can have a lean core for low memory environments but have an easy way to add a library that's blessed/secured by a trustworthy org.

for the scientific community, this group has been making good headway on a stdlib in javascript: https://github.com/stdlib-js/stdlib (I feel they deserve a shoutout here)

maybe instead of a single stdlib, an oligarchy of stdlibs for different communities' needs would be a useful approach (like what python tries to do)

Akamaozu commented 5 years ago

Saw this tweet a few minutes ago.

I think the linked discussion might be of interest to us

https://twitter.com/milessabin/status/1084783899677523968

markcellus commented 5 years ago

@Nimelrian

I think many people here misunderstand the goal and purpose of a standard library. It wouldn't replace entire frameworks. It would only supply well-tested, single implementations for common problems in our everyday use of the language.

Isn't this exactly what userland libraries aim to do? I personally believe that they already do a good job of that. I don't think people are talking frameworks here, we're talking userland libraries vs this new standard library. How are the potential libraries within this standard library any different from userland libraries that also exist to provide solutions "for common problems in our everyday use of the language"?

Another advantage would be that implementations can optimize the standard library functions using native code. That's impossible for userspace libraries.

I think this is an interesting idea and maybe this could be the focus of this group, but I certainly don't think that taking userland library features across multiple libraries and putting them into what we're calling a standard library is good for either the authoring teams or their consumers. Teams have spent years of hard work on their libraries and creating derivative works from them without their support is just rude. How do you plan to convince userland lib authors to support their work being exposed in this standard library vs their own repositories and under their own control?

I think @yacinehmito is right when he says

It would be great if people would define what they mean by standard library first if we want that discussion to ever move on. Wouldn't the opinions of everybody be heavily dependent on that?

This proposal needs to definitively address what it means by "standard library" or perhaps call out a few examples. Everyone has a different idea of what a potential "standard library" would include, so I think its important to make that clear--not in this comment thread--but in the actual proposal because it's still not clear to me, even after having read the README.

Pauan commented 5 years ago

How are the potential libraries within this standard library any different from userland libraries that also exist to provide solutions "for common problems in our everyday use of the language"?

They're a standard spec, thus widely implemented everywhere (cross-platform), so they can be relied upon.
They can be implemented to be faster/with less memory usage.
They do not need to be bundled with the user's code, thus reducing file sizes (a big deal with JavaScript).
They significantly ease prototyping (since you don't need to fiddle with npm, installing dependencies, bundling, etc.)
They significantly ease tweaking, debugging, and experimentation, since the features are available right in the Node/browser's dev console.
They make writing small shell scripts (or quick one-off scripts) dramatically easier.
They significantly ease teaching for beginners (since beginners can just jump right in and start writing code, without dealing with all the complexity of libraries and package managers right away).

Probably some other benefits that I wasn't able to think of.

but I certainly don't think that taking userland library features across multiple libraries and putting them into what we're calling a standard library is good for either the authoring teams or their consumers.

I don't think anybody is suggesting taking existing libraries (like lodash) and just copying them into the stdlib. The stdlib would be new, not a copy of existing libraries.

markcellus commented 5 years ago

No offense, but it sounds like you guys just want to create a new, authoritative platform of libraries that circumvent the strict processes of current standard's bodies by creating higher-level APIs based off of existing libraries in userland. You say these are "new" standard libraries, but you can't make a "new" library of "common features" that have already been built as npm packages, which are already used by much of the open-source community without it being "standard".

Everything you just mentioned is ~the same thing that existing userland libraries are doing~ great, but they are mostly claims to "make things easier" (I get it), but that's only a valuable solution if the current ecosystem is actually "hard", but I just don't think it is. So it just sounds like a solution to a problem that doesn't really exist--not at the magnitude at which you're emphasizing at least.

Overall, I just don't see how you're going to get people to disregard userland libraries for ones that you intend to expose here. All creating this "standard library" does is add another layer of red tape to ship software that is already available in the open-source community, which adds to the fragmentation. But this has been expressed already throughout this thread, so this will be my last comment unless I have something different to contribute.

evanplaice commented 5 years ago

Q: Rather than make new libraries that are built-in, should we standardize existing libraries and put them in the platform? These meet real needs, have adoption in existing code (so speedups will come sooner) and are more ergonomic.

:-1: Big no on this one. This is essentially taking the current popular libraries and 'blessing' them with special consideration. Performance speedups would be better served to all libraries if we had a common-reliable CDN that everybody actually used, better capabilities for caching, and possibly the ability to locally cache a post-parsed version of common libs.

I'd much rather see more emphasis on using public CDNs and providing LTS releases for popular libraries. This benefits everybody, not just the lucky few libraries that garner attention from TC39 (ie or whoever grants the blessings).

And let me be 'real' here. When you open up the potential for exclusivity, it's only a matter of time before the privilege becomes a 'pay to play' endeavor. It wasn't very long ago that the Node maintainers had to fork into IO.js to regain control of their project. It's no mystery that Joyent was actively blocking progress and selling influence to the highest bidders (ex MSFT, GOOG, etc) at the time.

Q: Should the platform focus exclusively on low-level capabilities, rather than building a standard library? The ecosystem does best at higher-level things, being more agile and not having to worry about rollout across browsers.

:+1: As much as I'm not in favor of a single common lib. I'm all for adding support for a few more low-level capabilities. I'd consider things like RegExp and Date to be standard features of any language and JS could potentially use a few more. Emphasis on very limited scope tho.

Q: How will built-in modules avoid the pitfalls of previous standardization efforts, which result in poor ergonomics, not really solving developers' needs, etc.?

It won't. For a language that is constantly adapting/changing to meet new demands, there's no way to accurately predict what will be required in the future (ex ES2030).

Bit rot is a very real issue. Standard library API's -- due to their ubiquity -- are essentially permanently frozen. Best case scenario, old versions are phased out on a predicable schedule to prevent breaking existing code. The sad fact is -- the more ubiquitous the API -- the harder it is to deprecate. Trying to force the ecosystem move on from obsolete code incurs a real cost (ie time/money).

Competition is Good

Falling in line with my previous comment, libraries like lodash are absolutely valuable to the community as a whole; but it's important to not forget that lodash is probably the 3rd-4th-5th successful iteration of a 'collection of functional operators' lib. It's great in large part as a result of competition. There will be more to come in the future.

From experience; at one point, one of my own libs achieved relative popularity for a while before it was eclipsed by a better alternative. What's good now will (ideally) be trash in 5 years. Not only is that a good thing for the overall quality of the lib ecosystem. It's a good lesson in humility for OSS devs. Even if you're a very talented dev; a rival project that provides quality documentation, is better marketed, is easier to use, is more inclusive to contributors, etc should win out in the long run.

If there will be a standard library, make it like any other JS library and see if it can reliably compete with the rest of the ecosystem first.

tc39 / proposal-built-in-modules

Should design a new standard library? #19