Add support for esm .remarkrc files

remcohaszing commented 3 years ago

Subject of the feature

I would like to add support for ESM .remarkrc files in remark-cli. :smile:

Problem

Since @wooorm is eager to ESMify (this is a verb now) a lot of packages, I believe remark-cli should support this, so users don’t run into compatibility issues when importing packages in their remark configuration files.

I believe this is a breaking change, because users may already have their build these configuration files. I’m not aware of any examples, but there’s probably 1 person on the world for whom this change would be breaking.

Considering this is already a breaking change and ESM files may import CJS files, I believe CJS support may be dropped.

This requires some upstream changes, I believe all of them are within the unified ecosystem.

Expected behavior

Users can create .remarkrc.mjs or .remarkrc.js files using ESM named exports.

I.e.:

import dictionary from 'dictionary-en';
import retextSpell from 'retext-spell';
import unified from 'unified';

export const presets = ['remark-preset-wooorm'];

export const plugins = [
  'remark-frontmatter',
  'remark-gfm',
  [
    'remark-retext',
    unified().use(retextSpell, { dictionary })
  ],
];

Alternatives

A default export could be used, but personally I prefer named exports.

remcohaszing commented 3 years ago

I’m willing to fix this by the way :smile:

wooorm commented 3 years ago

Some loose thoughts!

plugins && presets are essentially the same thing btw!

Some work would also be in load-plugin first, because we’d need to figure our what 'remark-gfm' points to. Supporting export maps, .cjs, .mjs, and package.json#type.

Then there’s work in https://github.com/unifiedjs/unified-engine/blob/53be1502aa05c6b4618402d3d9fa848d594739bf/lib/configuration.js#L19. We’d probably have to add .mjs and .cjs. We can detect whether .js is ESM or CJS based on the nearest package.json#type field.

And indeed, breaking. So we can also drop other features in the engine.

remcohaszing commented 3 years ago

Some more thoughts on this…

Currently undocumented is that remark-cli supports YAML based configurations as well. Should we keep this? I personally think JavaScript configuration files offer much more flexibility, i.e. the use of a custom unified processor for remark-retext.

If we could drop support for YAML, we could also drop support for string based plugin references, which removes the need to resolve plugins based on named exports. IMO this is too much magic anyway, although I get tools such as ESLint and Prettier do this as well.

import dictionary from 'dictionary-en';
import remarkFrontmatter from 'remark-frontmatter';
import remarkGfm from 'remark-gfm';
import { remarkMermaid } from 'remark-mermaid';
import remarkPresetWooorm from 'remark-preset-wooorm';
import remarkRetext from 'remark-retext';
import retextSpell from 'retext-spell';
import unified from 'unified';

export const presets = [remarkPresetWooorm];

export const plugins = [
  remarkFrontmatter,
  remarkGfm,
  // Not sure why one would do this, but this is just to show named exports may be used as well.
  remarkMermaid,
  [remarkRetext, unified().use(retextSpell, { dictionary })],
];

I do realize this is very verbose though. Optionally we could allow an array of promises in the plugins array, which would allow to write configuration files like this:

import dictionary from 'dictionary-en';
import retextSpell from 'retext-spell';
import unified from 'unified';

export const presets = [import('remark-preset-wooorm')];

export const plugins = [
  import('remark-frontmatter'),
  import('remark-gfm'),
  [import('remark-retext'), unified().use(retextSpell, { dictionary })],
];

If we do want to keep string based configurations, why not just use cosmiconfig? This is a popular package dedicated to solving this exact issue, used by for example Prettier and Stylelint. They already support YAML, JSON, and CJS configuration files and there’s a draft pull request for ESM.

Also if we want to keep support for this, I believe there’s value in supporting this for remark-retext as well.

import dictionary from 'dictionary-en';

export const presets = ['remark-preset-wooorm'];

export const plugins = [
  'remark-frontmatter',
  'remark-gfm',
  ['remark-retext', [['retext-spell', { dictionary }]]],
];

This also means we need to determine what a string plugin points to for ESM. Probably the default export.

Another though I was having is: What’s the goal of remark-cli? In my experience it’s a CLI for running a bunch or remark-lint-* whilst the remark-lint package is fairly useless (it just adds support for  comments).

In the beginning this was confusing to me. I just wanted to lint some markdown files, so why can’t I just yarn add remark-lint and run remark-lint from the command line? This is how ESLint, Stylelint, Prettier, and other linters I’ve used work. I also think this may be the main reason why markdownlint has more downloads than remark-cli.

Sorry if I got off topic too much. 😅

But I do think these considerations may affect how this is going to be implemented.

ChristianMurphy commented 3 years ago

Tangentially related with new formats being considered, it may be worth revisiting if https://github.com/davidtheclark/cosmiconfig or a similar config manager could be a good fit for the unified ecosystem.

wooorm commented 3 years ago

Currently undocumented is that remark-cli supports YAML

It’s documented in the engine docs, linked to in the remark-cli and rehype-cli readmes!

If we could drop support for YAML, we could also drop support for string based plugin references,

There’s still JSON. .remarkrc, .remarkrc.json, and package.json are all JSON.

which removes the need to resolve plugins based on named exports. IMO this is too much magic anyway,

ESLint, Babel, xo, postcss, etc, also all allow package.json or other JSON config files, to load packages through names. I don’t think we should drop that. I believe ESLint also always supported YAML. Anyway, I don’t care much for YAML.

Having a JS based config file is actually rather bad for caching, because well, it could do Math.random(), or other things to define the structure. That means the config can’t be cashed. And is the main reason why Parcel does not allow JS to define a config file. We could also instead ban JS and go all in on JSON.

If we do want to keep string based configurations, why not just use cosmiconfig? This is a popular package dedicated to solving this exact issue, used by for example Prettier and Stylelint. They already support YAML, JSON, and CJS configuration files and there’s a draft pull request for ESM.

Yeah we can switch to that. It wasn’t around, or at least in those tools, when I made remark-cli and the engine

Another though I was having is: What’s the goal of remark-cli? In my experience it’s a CLI for running a bunch or remark-lint-* whilst the remark-lint package is fairly useless (it just adds support for  comments).

In the beginning this was confusing to me. I just wanted to lint some markdown files, so why can’t I just yarn add remark-lint and run remark-lint from the command line? This is how ESLint, Stylelint, Prettier, and other linters I’ve used work. I also think this may be the main reason why markdownlint has more downloads than remark-cli.

Because unified is the only project that does both linting and compiling. That doesn’t exist in other languages, which are all fragmented. Rome is one solution that’s trying to fix it. I think unified has already solved all that while being modular instead of monolithic. All of my 500+ projects use remark-cli for both linting and formatting btw, and I quite like it 😅

remcohaszing commented 3 years ago

My preference is JavaScript (supports all values and allows comments) > YAML (better caching and allows comments) > JSON (better caching, but not comments)

I think this shows users have different preferences for this and we should keep support for JSON, YAML, and JavaScript, meaning we should add support for ESM.

I do believe cosmiconfig is the way to go. This means we should wait for cosmiconfig to support ESM, which they want to postpone until NodeJS 10 is EOL (2021-04-30)

wooorm commented 3 years ago

So it’s a two part thing.

ESM in config.js (this issue). We can do that with cosmicconfig on that date
ESM in plugins loaded from their string form name (e.g., remark --use plugin or from JSON/YAML). We could work on that earlier/later

wooorm commented 3 years ago

Second part is now done. Should we wait on cosmic config for the config files or get something in the engine for now (minor?)?

That leaves load-plugin as an optional thing, sort of outside the scope of this issue, but nice to have to actually match export maps?

remcohaszing commented 3 years ago

If this can be done easily, I’d say go for it.

jaydenseric commented 3 years ago

Note that some people are migrating from cosmiconfig to lilconfig, because it's lighter:

https://packagephobia.com/result?p=cosmiconfig@7.0.0 (5 dependencies, 743kB install size)
https://packagephobia.com/result?p=lilconfig@2.0.2 (0 dependencies, 15kB install size)

I haven't tried it myself yet, but something to keep in mind.

wooorm commented 3 years ago

@jaydenseric Interesting .Although, that does seem to miss ESM support (cosmiconfig does too, but at least is working on it)

ghost commented 3 years ago

Thanks @jaydenseric, lilconfig looks a lot better. @wooorm It looks like lilconfig follows major versions of cosmiconfig very closely. So when v8 lands, lilconfig should update accordingly.

JounQin commented 3 years ago

lilconfig is buggy and lack of maintenance than cosmiconfig.

https://github.com/antonk52/lilconfig/issues/17

wooorm commented 3 years ago

This was solved btw, forgot about this issue. And I kept with the existing custom loading—it’s pretty fast and works well and it was relatively fine to add ESM!

djmcgreal-cc commented 1 year ago

Hi, sorry for posting to an old thread but it's the closest I can find that might add context to my question! @wooorm, I'm too stupid to understand whether ESM configuration would allow me to override the cli's --use mechanism to look for an export other than the default. I'm trying to --use remark-code-import and it doesn't export a default.

wooorm commented 1 year ago

You can open new questions!

That’s a bug in that plugin. Plugins must export the plugin as the default.

djmcgreal-cc commented 1 year ago

Thanks for getting back to me so quickly :)

I posted here as I had an inkling some of the code snippets above were pertinent but I'm not a JS programmer and couldn't tell.

In this case I find that version 0.4.0 of remark-code-import works as I need so I'm good, and I've opened an issue in that repository for cli support.

remarkjs / remark