search

unifiedjs / unified

☔️ interface for parsing, inspecting, transforming, and serializing content through syntax trees
https://unifiedjs.com
MIT License
4.47k stars 110 forks source link

Next major for the ecosystem #121

Closed wooorm closed 3 years ago

wooorm commented 3 years ago

Soon (April), it’s time to switch to ESM (without CJS backup) for all packages in the ecosystem. That’s a lot of projects switching over, and a round of majors that bubbles from the bottom through to the top.

I’ve had some experience with that recently in a push in micromark (with CJS fallback, causing issues). My recent own projects dioscuri and xdm are fully ESM. It’s going to be a lot of fun for the ecosystem 😅

Otherwise, here are some changes slate for this next bubbling:

Big breaking changes:

Small breaking changes:

For ES++ features (https://github.com/unifiedjs/rfcs/pull/4), I’d say we can do that after we switch to ESM, because ESM will already be a lot of work, and it gives us a baseline of what ES features / engines to support.

chrisrzhou commented 3 years ago

Are there discussion threads (any links to notable discussions are good enough!) where I can read more into the motivation to move formally to using ESM? Inquiring to stay up to date with the unified ecosystem, and also getting context in the broader JS ecosystem.

wooorm commented 3 years ago

https://github.com/micromark/micromark/pull/27 and other PRs there, but that was both ESM and CJS. See also https://twitter.com/sindresorhus/status/1349294527350149121. The motivation is essentially: we’ll have to move some time. This is a good time!

nnmrts commented 3 years ago

Great to see unified switching over as well!

@wooorm Please don't archive retextjs/retext-sentiment. I'm ready to maintain it if you want. It just seems too useful to me to just forget it.

wooorm commented 3 years ago

I’m also going to try to add jsdoc based types to everything (starting with ±150 of my own projects).

Another big one is to add support for plugins as ESM in unified-engine: the only way forward I see is if plugins export default? Or have a named plugin export? 🤔 I prefer the first from the unified-engine perspective, but as an author I don’t like default exports.

chrisrzhou commented 3 years ago

@wooorm, (for my own learning), a few questions relating to his upcoming refactor:

  1. What are the benefits for jsdoc types vs .d.ts vs strictly implementing the code in TS?

The main benefits I'm inferring is decoupling the dependency and reliance on TS, so that if the ecosystem needs to be free of TS in the future, we at least have JSDocs with an easy path to decoupling in the future? I'm curious about the decision because I'm facing similar concerns with trying to keep source code as independent from tooling/dependencies as possible (flow was king at one point, typescript is currently 'king', but no one knows what will happen with the typing ecosystem in the future).

  1. With the focus on ES modules, it seems pretty amazing that we don't need build tools since it's natively supported by Node and modern browsers. How do you resolve dependencies in unified ecosystem that are not migrated to ES modules? E.g. we cannot naturally rely on package.json's type: module if an upstream dependency hasn't formally migrated to ES module (even if the package supports ES modules through build tools and exporting "module: index.module.js" in package.json, Node scripts won't treat it as a formal ES module). Is this problem avoided in the ecosystem because the ecosystem has very limited external dependencies?

  2. Related to 2, should packages (even microutils) in the ecosystem care about bundling/minifying or should that just be left to upstream consumers?

Would love to hear the perspectives and approaches of the team here, so I can also learn something from it!

And my (personal and likely not-important) opinion on:

Another big one is to add support for plugins as ESM in unified-engine: the only way forward I see is if plugins export default? Or have a named plugin export? 🤔 I prefer the first from the unified-engine perspective, but as an author I don’t like default exports.

I think most authors avoid default exports and use named exports (explicit named export is useful as a static guard, also great for refactoring), although I think that if we have to consider how consumers would use an API, we should make the appropriate decision based on how this library is intended to be consumed. For myself, I use 1) named exports for internal implementation all the time and 2) decide the final public export based on what makes sense for the consumer.

chrisrzhou commented 3 years ago

Didn't realize Github does not support comment threads, so it's kind of annoying keeping track of discussions and responses :P (this comment further adds to the problem)

wooorm commented 3 years ago

What are the benefits for jsdoc types vs .d.ts vs strictly implementing the code in TS?

I strongly dislike having to compile things. I think the benefit of JS is that it just runs everywhere. If you’re going to compile, IMO there are much nicer languages. I don’t write these projects in JS because I like JS per se (although I do), I do it because it works everywhere. And, these projects are all long lived: I’ve been maintaining most stuff for 7+ years already. Rewriting the whole ecosystem for every new hype is a lot of work.

This is also why is dislike “modern” javascript. One such example of this is how the projects that’ve pushed ESM over the years, actually don’t support ESM. Webpack, snowpack configs, Jest: they all don’t get actual ESM. I’m not adverse to types per se, and especially would like to use their information for docs. But it’s rather complex to try to make a single, low-config, doc system based on them that works nicely for all my projects, so maybe that’s for the next iteration?

We (@ChristianMurphy and I) did run into a couple places where JSDoc based TS doesn’t cut it yet, though. And while some of it might be me being a TS beginner, I do have some doubts. It’s adding a lot of complexity for my style of small, well-tested projects, but not adding a lot of benefits.

So, you’re seeing some of the same things I’m seeing, but I hope the above explains a bit more about my current state of reasoning

With the focus on ES modules, it seems pretty amazing that we don't need build tools since it's natively supported by Node and modern browsers.

Omgosh! It is! Of course, bundling would be needed in most professional settings. But it works. And it works nicely.

How do you resolve dependencies in unified ecosystem that are not migrated to ES modules

Using ESM from CJS is a bit hard, you have to use an import(x) expression. The other way around is smooth sailing: you can import CJS and ESM mostly seamlessly from ESM.

Node scripts won't treat it as a formal ES module

You can set "type": "module" in a package.json and everything under it will be seen as ESM, and CJS otherwise. Or you can opt-in/out by using .mjs or .cjs extensions.

should packages (even microutils) in the ecosystem care about bundling/minifying or should that just be left to upstream consumers?

Can you expand on this? IMO, all projects that are “universal” (node + browser) should care about size. But if you’re asking about dual-CJS/ESM, IMO it’s too much work and too buggy to try it. Pick ESM.

For myself, I use 1) named exports for internal implementation all the time and 2) decide the final public export based on what makes sense for the consumer.

I agree on not using default exports for libraries: while names are a bit superfluous in most of my projects (as they export one thing), default exports have downsides. But plugins have to be importable by tools too. So it has to be either a single named identifier (plugin?) or the default export. Using plugin for every plugin is just as well a bad experience for users that do import it manually. So that’s not my preference. Babel and Rollup do support ESM plugins nicely, and they use default exports, so there’s some precedence for it too!

chrisrzhou commented 3 years ago

Thanks for the detailed response, it helps affirm some of the dilemmas I've been having around 'modern' JS.

I 100% agree with a lot of the statements you (for rest of the post, "you" refers to the unified team! ) made about compiling, typing, and in general coupling standard JS source code with build tools. My experience with JS has largely been a follower and consumer of such tools, and only recently by following and studying code in various repos (e.g. unified, three, tape), did I start paying attention why authors take on the cost of maintaining 'buildless' and 'typeless' libraries.

On typing, TS does offer a much delightful developer experience, but I think the danger is the community being over-eager in thinking that TS guarantees type-safe code. TS doesn't protect against runtime invalid types, and in the end it just boils down to a simple question if consumers are using the APIs as-intended. I personally think TS improves the developer experience and exploration of code, but I think that is as far the value I can see with it, and I agree with your principles on avoiding rewriting libraries if hyped-package@will.eventually.be.outclassed actually gets in the way of authoring source code. I think the JSDoc with TS typings is a safe decoupled approach that you can toss out TS and still keep readable documentation (I'm assuming unified's JSdoc strings are sourcing from TS types?).

On build tools, babel and webpack etc are powerful and have allowed developers to write basically any syntax and rely on transpiling to arrive to functioning JS. This has allowed developers to be expressive, and caused a huge momentum and explosion in the JS ecosystem, but I also tend to see this as a deviation from specs/standards-driven development. In the end, the real standards are those of the language and ecosystem (browsers). Everything else is a standard on top of another standard, and none of these will last. We've moved from grunt- > gulp -> webpack -> rollup -> snowpack -> something else but we're really just playing on top of the base standard of writing code that 'just works'. Authoring code with new syntax sugar is sweet (pun intended), but just as coffeescript eventually fell out, so would any source code authored in potentially-non-compliant ECMA proposals fall out too. And this all deteriorates to teams having no choice but to author code in an outdated/incorrect standard, or basically having to push the "refactor everything" red button at some point. The cost of delightful development is eventually paid with a rewrite, something that you've alluded to.

Anyway, the above is just myself chatting and responding to something that I've slowly come expose to and appreciate because of the discussions and journey in learning unified and its implementation. Back to the specific questions that I think I may not have expressed very well, so structuring it a little better:

Question 1: If I were writing ESM packages intended for ESM consumers, should I care about exporting a .cjs export, and also care about minified exports or should I leave it to consumers to decide how to handle this? Will unified ecosystem export .cjs bundles, minified bundles etc, which would basically means that there is a compile/build process for non-ESM outputs? Or is unified ecosystem going to pure and completely compile-/build-free which means limiting the ecosystem to ESM-only consumers?

Question 2: (see below code snippet for detail) If my package depends on an NPM dependency that isn't written in standard ESM, but does provide an ESM output, this would not work with node ./my-esm-script.js. Does unified packages face this problem or it is irrelevant because the ecosystem is self-contained (no external dependencies)? If this problem exists for unified, how can this be solved?

my-esm-script.js

import x from 'vendor-library';

x();

vendor-library/package.json

 {
  "module": "dist/index.module.js",
  "main": "dist/index.js",
 }

Note that there is NO "type": "module" field because the author of vendor-library did not set it up that way since it wasn't a library developed as a pure ESM module, but rather the final dist files are generated through some build tool (e.g. microbundle, rollup etc).

Running node ./my-esm-script.js will throw an error because

In the above example, I cannot control setting "type": "module" in vendor-library/package.json because the vendor owns the implementation. I believe that unless the vendor library follows the ESM standards, there is no way I can consume this vendor package in a pure ESM way (without build/compile steps)?

nnmrts commented 3 years ago

In the above example, I cannot control setting "type": "module" in vendor-library/package.json because the vendor owns the implementation. I believe that unless the vendor library follows the ESM standards, there is no way I can consume this vendor package in a pure ESM way (without build/compile steps)?

I went on a bit of a tangent here, based on a misunderstanding of your second question, @chrisrzhou. Scroll down to see my attempt to answer that and the first one. I feel you and that's the one big thing I'm always unsure on how to solve it. I really like how the ecosystem and community in general is finally but slowly moving to the methodology you described, the somewhat naive philosophy to at least try to write "future-proof" code. To follow the standard, to do things that work everywhere. Well at least everywhere a developer expects it to work. But, yes, you are totally right, at some point, you also just have to accept that projects progress with wildly different velocities and manners to that desired pure ESM utopia. Some will even never get there, because they are either unmaintained, or the respective maintainer doesn't believe in that already mentioned philosophy. People have different backgrounds and expectations of how code should be interacted with, what software should do and what it shouldn't do, and that's fine. From a personal perspective, for example, I would never use Typescript in the context of coding modules and projects that are meant to be used by other fellow developers, and I don't support or share the philosophy behind the language at all. People can shout at me all they want about its "benefits", I know about them, I worked with Typescript professionally, but as long as it isn't (and shouldn't be) directly interpreted by browsers and runtimes and it still needs to be compiled, as long as it just follows a standard that's ultimately controlled by a single company instead of at least a couple of companies, important developers we kind of trust and a transparent well-thought-out process of introducing new features, it just makes no sense to me why I should use it in a community context. But without going to much into depth about all that, I don't think the majority of unifieds modules depend or have to depend on such CJS `vendor-libraries` you mentioned. I didn't go through all of the hundreds of `package.json`s, but as far as I know, most of them are themselves that `vendor-library` you will eventually depend on, the fundamental building blocks for bigger projects.

Question 1:

Most of unifieds projects, correct me if I'm wrong, also aren't meant to be used directly in a browser, they are meant to be used by other developers and their projects. And I understand the accessibility concerncs, the aspiration of deploying your project in all imaginable formats to please the most amount of developers possible. But the answer to accessibility ends where the question of complacency starts. I think @wooorm ultimately decides which way unified will go, but I definitely think it, and - trying to answer your first question - anyone writing "ESM packages intended for ESM consumers", should "leave it to consumers to decide how to handle this".

Question 2:

Your second question has multiple layers though. If we are talking about Node.js and the general context of unified in its current state, no, there aren't any conflicts.

Running node ./my-esm-script.js will not throw an error, importing CJS in ESM (in Node.js) is totally fine. See this super simple example: https://github.com/nnmrts/esm-test which imports https://github.com/nnmrts/cjs-test. But yes, this won't import the ESM format file, the index.module.js. It will use the usual, main CJS index.js file. If that is what you are referring to, true, you won't get the whole ESM "experience", but does that really matter? Your own code is still in ESM. So no, unified doesn't and never will face a "problem" here, as long as it sticks to Node.js.

If you are referring to the more general context of "real" ESM, independent of Node.js, then yes, there would be an error if you import the cjs file. Since ESM also doesn't allow you to just do import x from "vendor-library"; (you have to refer to an actual file), you would ideally refer to the ESM format file from that vendor-library directly, like:

import x from "vendor-library/dist/index.module.js";

Or actually, in this case:

import x from "./node_modules/vendor-library/dist/index.module.js";

Do I personally believe unified (and similar projects for what it's worth) should be Node.js-independent and use the "real" ESM format of importing actual files, maybe even by using Deno and publishing to nest.land? Yes absolutely. Do I think it would be healthy to do this at the same time it switches to ESM syntax and output in general? Nope. I would suggest to focus on that first, to make migration easier and especially since "pure" ESM, Deno, nest.land and all that stuff is still pretty new territory for most people.

wooorm commented 3 years ago

@chrisrzhou

If I were writing ESM packages intended for ESM consumers, should I care about exporting a .cjs export, and also care about minified exports or should I leave it to consumers to decide how to handle this? Will unified ecosystem export .cjs bundles, minified bundles etc, which would basically means that there is a compile/build process for non-ESM outputs? Or is unified ecosystem going to pure and completely compile-/build-free which means limiting the ecosystem to ESM-only consumers?

I believe source code should be readable. I don’t see a lot of value in publishing minified bundles. Dual packages, so both ESM and CJS, are again complex, but importantly also come with quite a hazard.

There are cases where, based on the ecosystem (browser, node, react-native, etc), or based on other environmental things (dev vs. production), different code should run. These “conditions” are being added to Node. In my RSC demo I found that the React team was experimenting with them for server vs. client. These conditions are probably exceptions though.

If my package depends on an NPM dependency that isn't written in standard ESM, but does provide an ESM output, this would not work with node ./my-esm-script.js. Does unified packages face this problem or it is irrelevant because the ecosystem is self-contained (no external dependencies)? If this problem exists for unified, how can this be solved?

The module field shown here is not “standardized” by Node, indeed. It’s an instruction for faux-ESM bundlers such as webpack.

I suggest for folks that maintain semi-inactive projects/apps to stick with what they have and not update dependencies moving to ESM. After a couple of warts in Jest/Electron/webpack/Next/CRA are solved, hopefully in a few months, I think it’s fine for folks of semi-active or active projects to move to ESM: they can update their dependencies, and use CJS packages just fine.

So, for “Does unified packages face this problem”: No, unified does not face such a problem, because switching over to ESM solves everything.

@nnmrts

I think @wooorm ultimately decides which way unified will go

I do decide a lot through sheer activity and also some level of seniority in this ecosystem, but it’s a democratic process: https://github.com/unifiedjs/collective/blob/main/decisions.md

And interesting for you both: in the future we’ll get import maps in browsers: https://github.com/WICG/import-maps. That allows these “bare” specifiers (package, package/file.js, @scope/page) to work there too. I’m imagining that a tool will be made to generate such an export map from node_modules/, so that at dev-time it work just as seamlessly in a browser as on Node, and then everything is bundled together for production.

EDIT: And, the inverse is also being worked on (but experimental) in Node, with Node loaders. Here’s how to import http:// and such in Node: https://github.com/node-loader/node-loader-http

chrisrzhou commented 3 years ago

Seems like a really disruptive/hectic period for JS developers, but hoping the equilibrium settles to a simpler and standard ESM with less build tools. Would be great if more JS code is just standard JS code! Thanks for all the detailed response and info!

aminya commented 3 years ago

For the packages that have other dependencies, and there is no transpilation happening, we cannot use Pure ESM packages because:

wooorm commented 3 years ago

The above is incorrect because:

talentlessguy commented 3 years ago

Sorry if this question is unrelated but does moving to ESM also mean switching to ES6+? Or unified core package will remain ES3 (or any other pre-ES6 version)

wooorm commented 3 years ago

👋

Why are you asking?

Some newer features are/will be used other than just the import/export syntax, yes, such as classes. My opinion remains that var and named functions for example are still as valid in ES++ as in ES1. See the last paragraph in the original post for a reference to more discussion on this!

wooorm commented 3 years ago

Does anyone have experience with Deno / Skypack / jspm and such? Would be nice to document support for those, if it works, in the install section next to npm. E.g.,

Deno/browser:

```js
import {visit} from 'https://cdn.skypack.dev/unist-util-visit'
talentlessguy commented 3 years ago

@wooorm

I think esm.sh deals with it better because it also provides type declarations for Deno. And it also has parameters for more concise setup (e.g. pick ES target / bundle css / pick bundle mode)

import {visit} from 'https://esm.sh/unist-util-visit'

console.log(visit)

afaik Skypack doesn't do that

wooorm commented 3 years ago

Apparently you can add ?dts to Skypack. Then it’ll also add a x-typescript-types header. E.g., https://cdn.skypack.dev/unist-util-visit?dts. Which is what Deno says they support: https://deno.land/manual@main/typescript/types 🤔 Deno seems to suggest Skypack and doesn’t mention esm.sh. So both seem to be rather similar?

talentlessguy commented 3 years ago

@wooorm oops my bad, yes Skypack also provides types (but you have to enable it explicitly which I don't like tbh, better have types fetched by default)

Another difference is this:

Different with Skypack and jspm, esm.sh will bundle all dependencies(exclude peerDependencies) for each package, that gives more better loading speed.

SalahAdDin commented 3 years ago

It is not clear for me how we will replace remark plugins by rehype plugins if the first ones works for markdow, what we want to parse, and the second ones works for html.

ChristianMurphy commented 3 years ago

@SalahAdDin the plugins noted in "archive and deprecate remark plugins that (should) have rehype equivalents, and add some docs somewhere on whether folks should use remark or rehype plugins (currently quite confusing) "

Are ones which work on HTML/with HTML. Your question is a good one, and highlights where the confusion comes in.

Here's an example of where things can go wrong trying to do things from the markdown side. Take, remark-external-links as an example, it adds target and rel attributes to links out to external sites. Given some content:

[markdown link](https://example.com)

<a href="https://example.com">html link</a>

Where do you expect the rel and target attributes to go?

One may think on both links, since they are both external. But with remark-external-links it only does the first https://codesandbox.io/s/remark-rehype-debug-external-links-tfnlz?file=/src/index.js The second is not part of the markdown, it's part of an embedded HTML document. This can be unexpected and confusing to many adopters.

Making this a rehype plugin resolves this, with a rehype plugin both links would be updated.

The other plugins listed also have the same or similar confusion. They kinda work thanks to hName and hProperties https://github.com/syntax-tree/mdast-util-to-hast#hname But they only work on the markdown part of the document, which is not what most people want/expect.

SalahAdDin commented 3 years ago

@SalahAdDin the plugins noted in "archive and deprecate remark plugins that (should) have rehype equivalents, and add some docs somewhere on whether folks should use remark or rehype plugins (currently quite confusing) "

Are ones which work on HTML/with HTML. Your question is a good one, and highlights where the confusion comes in.

Here's an example of where things can go wrong trying to do things from the markdown side. Take, remark-external-links as an example, it adds target and rel attributes to links out to external sites. Given some content:

[markdown link](https://example.com)

<a href="https://example.com">html link</a>

Where do you expect the rel and target attributes to go?

One may think on both links, since they are both external. But with remark-external-links it only does the first https://codesandbox.io/s/remark-rehype-debug-external-links-tfnlz?file=/src/index.js The second is not part of the markdown, it's part of an embedded HTML document. This can be unexpected and confusing to many adopters.

Making this a rehype plugin resolves this, with a rehype plugin both links would be updated.

The other plugins listed also have the same or similar confusion. They kinda work thanks to hName and hProperties https://github.com/syntax-tree/mdast-util-to-hast#hname But they only work on the markdown part of the document, which is not what most people want/expect.

So, it means rehype plugins handles both markdown and embebded html, right?

ChristianMurphy commented 3 years ago

Yes, when remark is used with remark-rehype and rehype-raw it does both

SalahAdDin commented 3 years ago

Yes, when remark is used with remark-rehype and rehype-raw it does both

Ok, so, having rehype-raw is fair enough to render markdown properly.

Thanks.

ChristianMurphy commented 3 years ago

remark-rehype is the plugin that translates markdown to HTML. rehype-raw goes though any remaining unparsed/embedded HTML documents and parses those as well.

SalahAdDin commented 3 years ago

remark-rehype is the plugin that translates markdown to HTML. rehype-raw goes though any remaining unparsed/embedded HTML documents and parses those as well.

We hope it will be included in the documentation, :')

wooorm commented 3 years ago

Could you expand on what documentation you want to see? E.g., the readme for remark-rehype says precisely that: https://github.com/remarkjs/remark-rehype#remark-rehype

205g0 commented 3 years ago
  • From ESM, you can import CJS without any problems: import whatever from 'some-commonjs-package' works

@wooorm, are you sure? I just tried to import express into into an ESM code base and compile with esbuild index.ts --outdir=dist --format=esm --platform=node --bundle, with both esModuleInterop true/false and get still errors (FWIW, Error: Dynamic require of "events" is not supported).

wooorm commented 3 years ago

This seems off topic here as it’s not about unified projects or the upcoming majors? Your question mentions esModuleInterop which I don’t think is an esbuild option? Maybe better asked somewhere else.

205g0 commented 3 years ago

@wooorm apologies I wasn't clear: you stated that you can import CJS from ESM which I think is not true. And I gave just an example with one of the most popular node libs. You can try this example (import express into an ESM code base) also with tsc which will yield the same result: It won't work.

Happy to get your view and a better understanding why you think that you can generally import CJS to ESM. This is pretty new to me.

PS, since unified's full shift to ESM and because unified is used for a significant part with node and the node ecosystem hasn't been fully transformed to ESM (e.g., express) I replied to this thread.

ChristianMurphy commented 3 years ago

@205g0 it is possible to import CJS from ESM https://nodejs.org/api/esm.html#esm_interoperability_with_commonjs explains how interoperability works. https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c offers advise on how to configure TS to support ESM, this thread is also can good place to discuss supporting ESM in various build tools, though for ESBuild specific support reaching out the ESBuild community directly may be a good idea.

205g0 commented 3 years ago

@ChristianMurphy Thanks for chiming in and sorry again for going back to this ESM discussion. Your links are great resources and a good start into interoperability. Still, if you follow all the guidelines from your links you'll hit edge cases such as the dynamic requires within express which you can't work around. I wished it was different.

ChristianMurphy commented 3 years ago

@205g0 There are several migration paths suggested in the guide (https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c). The very first option:

Use ESM yourself. (preferred)

Seems to work well with express. For example https://stackblitz.com/edit/http-server-v39qgo?file=server.js, shows that express and micromark coexist happily.

This thread isn't really the place to try to debug your specific app. support channels could include:

205g0 commented 3 years ago

I hesitated to reply because of all the OT markings and as you—I do not need another discussion wasting our time. But your Stackblitz example is JS not TS, this does not work with TS. You could now say that @wooorm's initial statement is technically right or that this is TS' problem not yours...

But the topic is actually unified's full move to ESM and how the users might deal with it. And since this thread is about the general roadmap of unified I politely hint you contributors that a large part of (TS) users run into problems. This might change in one year but yeah.

Now, it's your call how to deal with this info, treat me like a noob ("ask a support channel"), OT my posts, ban me, silence me. Up to you, I am fine with anything and I just wanted to help and was not looking to get help and not to debug my specific app. Have a great day. 🙂

SalahAdDin commented 3 years ago

I hesitated to reply because of all the OT markings and as you—I do not need another discussion wasting our time. But your Stackblitz example is JS not TS, this does not work with TS. You could now say that @wooorm's initial statement is technically right or that this is TS' problem not yours...

But the topic is actually unified's full move to ESM and how the users might deal with it. And since this thread is about the general roadmap of unified I politely hint you contributors that a large part of (TS) users run into problems. This might change in one year but yeah.

Now, it's your call how to deal with this info, treat me like a noob ("ask a support channel"), OT my posts, ban me, silence me. Up to you, I am fine with anything and I just wanted to help and was not looking to get help and not to debug my specific app. Have a great day.

Come on man, Christian is a very cool guy who always is willing to help you with everything, even with small things! I think we could move this conversation to the discussion section since Typescript is a very importan issue for many developers, me included, but we don't want to deviate the main purporse of this pull request.

What do you think?

205g0 commented 3 years ago

@SalahAdDin, thanks for deescalating the conversation and yes, Christian is cool but I can't follow you, the discussion around ESM was a significant part of this thread.

benrbray commented 3 years ago

I want to chime in here and say that while I really love using Remark and appreciate the work put into it, getting the ESM modules to work TS is an absolute pain. I get wanting to move forward with an ES2015 feature that has had over six years now to catch on, but the simple fact is that the rest of the ecosystem still isn't ready. Until then, I really wish that Unified would change its mind about the decision not to include backward-compatible packages.

I'm using remark in an app built with TS + webpack + electron. With this combination of tools, some of the unified packages (mostly, for tree manipulation) simply don't work. Instead, I ended up either copying the unified source files manually into my own tree and rewriting the imports, or rewriting the functionality myself in pure TS (the unified API currently likes to wrap everything up in all-but-the-kitchen-sink functions, which are difficult to write types for, as evidenced by the current JSDoc strings).

On a positive note, I'm glad that unified is flexible enough that I can write my own solutions when needed. I just wish I didn't have to.

wooorm commented 3 years ago

Folks, it’s fine to talk about ESM. Either here or in other places. But this issue is about moving to ESM and other things in majors. And the discussion on whether to do it or not was had in Feb and March. Christian and I are trying to keep the thread manageable by hiding stuff that is irrelevant for the majors while still answering questions and pointing towards better places to discuss ESM.

@benrbray I don’t think dual ESM/CJS is viable for the JavaScript ecosystem: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c#gistcomment-3851022.

Re TS: one part of the majors is also that everything is (strongly) typed, externally but also inside projects themselves. So that at least should help you a lot when you can move to ESM in TS.

ChristianMurphy commented 3 years ago

Stackblitz example is JS not TS, this does not work with TS

@205g0 it does work with TS, here it is with TypeScript checking https://stackblitz.com/edit/http-server-quvryg?file=server.ts (note that stackblitz's editing UI is not in sync with TSC on what is an error or not, the CLI shows actual errors) The guide at https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c provides assistance how how to configure TypeScript, it was followed for this stackblitz and works nicely.

am fine with anything and I just wanted to help and was not looking to get help and not to debug my specific app.

I want to help as well. You've said specific build tools can't work on/with ESM, each you have shared so far can work, I'm trying to point you toward the solution(s) without derailing the main discussion.

ChristianMurphy commented 3 years ago

I'd second @wooorm's comment in https://github.com/unifiedjs/unified/issues/121#issuecomment-896650171 and would add:

I'm using remark in an app built with TS + webpack + electron. With this combination of tools, some of the unified packages (mostly, for tree manipulation) simply don't work

It may be worth opening a discussion to talk through this (https://github.com/unifiedjs/.github/blob/main/support.md#questions). With https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c#im-having-problems-with-esm-and-typescript and https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c#how-can-i-import-esm-in-electron it should be possible to run the latest version on Electron.

I ended up either copying the unified source files manually into my own tree and rewriting the imports, or rewriting the functionality myself in pure TS

In the latest version all utilities are pure TypeScript. TypeScript offers two syntaxes, annotation syntax and JSDoc syntax, unified primarily uses the latter. (https://github.com/unifiedjs/rfcs/pull/5)

205g0 commented 3 years ago

Guys, you are great, you really try to help. Nice and highly appreciated. @ChristianMurphy Thanks for going down the TS rabbit-hole and proving that it's working, nice. However, the reality is that, yes with tsc you get it working, you need to fiddle a bit here and there and congrats. Once your build a real-world app with many libs, project references, stuff gets complicated, edge cases, weird error messages. Worse, when you use real build system (fwiw, tsc is not one and never meant to be one) such as esbuild which beats everything out there, is 1,000,000x faster and is getting the defacto standard—importing CJS into ESM code just doesn't work anymore with --platform=node and --format=esm. Doesn't matter how much you fiddle and tinker. Is esbuild a hype or fad? Maybe but others using it as the foundation, such as vite. So yes, it's a significant part of the ecosystem and the creator Evan is one of the most talented dev I know. I haven't tried yet all other build systems out there but would expect similar nuances just breaking code.

So, I'm not gonna prove you are wrong or I am right because it's would be just about who is right or the smarter guy in this thread and so on. Plus, there is no away around ESM, that's the future and either we get along with ESM, or switch to Deno or to Go or whatever haha.

I think it's just important to remind maintainers about real pains and what we do, such as @benrbray did and allow discussions and be careful with any kind of post modding (e.g., OT marking, even if there were no bad intentions). There're even libs where maintainers even do not close solved issues but always let the issue author close or keep them open. IDK if I would do that but just to show how cautious they are with any kind of communication which might offend.

NGL, I was very surprised when I read @wooorm's stance on TS vs JSDoc and yeah. I don't want to go down that rabbit-hole of a debate, either but let's just put pros and cons aside (I mean I was like @wooorm and thought for more than one year that TS is kind of useless until I thought the exact opposite—or that TS offers the most advanced and most responsive type system of ALL langs BY FAR providing tons of benefits making it hard to go back, but again this is another topic), the point is—

Maybe TS is a fad or hype (which I find very unlikely) the majority of users is on TS or what I would call "all pro projects". There are still projects in JS or JS with JSDoc but this is legacy code, except some of course, so yeah. What I want to say, if you guys would be on TS, maybe there wouldn't be a solution to this esm drama, either but your understanding for your users would be probably better. Maybe also your creativity and who knows, maybe there is some solution or work-around which is better.

Whatever, thanks for taking the time discussing with me, providing help and maintaining this great piece of work for such a long time. 🙂

ChristianMurphy commented 3 years ago

importing CJS into ESM code just doesn't work anymore with --platform=node and --format=esm. Doesn't matter how much you fiddle and tinker.

esbuild does work, with TypeScript and with the exact options you mention, here is a minimal reproducible example https://gist.github.com/ChristianMurphy/761ce0eeba33e243843665db00eb380e npm start builds and starts the server as expected. Again, if you encountering a specific error with esbuild, I'd reach out to the esbuild community, I've found Evan and the community to be responsive and insightful.

Once your build a real-world app with many libs, project references, stuff gets complicated, edge cases, weird error messages if you guys would be on TS, maybe there wouldn't be a solution to this esm drama

I manage several large production apps, several of which currently use TypeScript, several are plain JS. I am aware of how complex upgrades can be, however in my experience upgrading to ESM was switching a few build options, and updating require to import. If issues do crop up, the support channels for specific build tools are excellent resource to get pointed in the right direction.

It's also worth repeating https://github.com/unifiedjs/unified/issues/121#issuecomment-896954625, unified and most of its utilities are plain JavaScript with JSDoc that are TypeScript checked in the latest major. Both JS and TS remain compatible with Unified.

ChristianMurphy commented 3 years ago

Now, it's your call how to deal with this info, treat me like a noob ("ask a support channel"), OT my posts, ban me, silence me. Up to you, I am fine with anything and I just wanted to help and was not looking to get help and not to debug my specific app.

I think it's just important to remind maintainers about real pains and what we do, such as benrbray did and allow discussions and be careful with any kind of post modding (e.g., OT marking, even if there were no bad intentions). There're even libs where maintainers even do not close solved issues but always let the issue author close or keep them open. IDK if I would do that but just to show how cautious they are with any kind of communication which might offend.

Support channels are for people of all experience levels, experienced developers have questions too.

Off topic marking is not meant to offend. Unified has clear moderation guidance https://github.com/unifiedjs/collective/blob/HEAD/moderation.md keeping a conversation on topic is within that guidance.

The discussing ESM is within this conversation. Comments like https://github.com/unifiedjs/unified/issues/121#issuecomment-896642211, which respectfully express frustration on the topic on hand are fine. Rants diving into unrelated topics, making claims which have directly been shown to be false, and attacks on the maintainers of projects are not welcome.

I hear your frustration @205g0, I respect your view that ESM migration can in some situations be difficult. Please share it in a way which is respectful to the conversation at hand and the members involved. And as you learn about ESM and migrating to standard import syntax, consider sharing challenges you've encountered and how you've solved them in threads like https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c so that members of unified and other communities can benefit.

/cc @unifiedjs/moderators

205g0 commented 3 years ago

esbuild does work, with TypeScript and with the exact options you mention

@ChristianMurphy, nope, esbuild does not work when importing CJS modules such as express into a ESM code base. You missed the --bundle option (it was mentioned in my very first post but not my last one, my bad). FWIW, since esbuild cannot transpile imported files when not bundling nor transpiling all files in a folder[1], bundling, even on the backend is kind of the default choice. This is not a support post.

[1] You could but need custom scripts on top of esbuild.

ChristianMurphy commented 3 years ago

@205g0 Again it is possible and does work, including with --bundle, this very question/concern has been addressed by Evan in the past https://github.com/evanw/esbuild/issues/1232#issuecomment-830677608 Here is an example of it working https://gist.github.com/ChristianMurphy/db8a1dff8cdf47d315d49edb5d480d3e

This is not a support post

This would go a lot faster if it was. Directly asking how to make it work, and providing context would be much easier for both of us. Rather than the back and forth of you blaming various tools for "not working" and others showing that they do in fact work.

wooorm commented 3 years ago

Done! Alright, folks, everything is now ESM and typed! The packages up for deprecation were deprecated. I’ve “soft deprecated” certain packages by first updating them and then marking them as legacy, like so, to prevent too much churn. They can be fully deprecated later but this makes it a bit easier to still land security fixes when needed. I’ve also refrained from a couple of other changes such as depth -> rank for mdast headings.

If you have questions about ESM, you might find answers in this Gist. Generally, the problems will be with build tools, bundlers, and site generators, so you might also raise questions or find answers in their corresponding issue trackers or discussion channels. If your question relates to unified packages, feel free to open a discussion with a reproduction in one of our boards.