remarkjs / remark

markdown processor powered by plugins part of the @unifiedjs collective
https://remark.js.org
MIT License
7.68k stars 357 forks source link

remark-cli and .remarkrc.js throws "Only file and data URLs are supported by the default ESM loader" #786

Closed Zamiell closed 3 years ago

Zamiell commented 3 years ago

Error Message

On the latest version, remark no longer works properly on Windows, resulting in the following error:

README.md
  1:1  error  Error: Cannot parse file `.remarkrc.js`
Only file and data URLs are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'd:'
    at FormattedError (D:\Repositories\remark-test\node_modules\fault\index.js:29:12)
    at onparse (D:\Repositories\remark-test\node_modules\unified-engine\lib\find-up.js:152:13)
    at done (D:\Repositories\remark-test\node_modules\trough\wrap.js:55:16)

× 1 error

This is a regression because it worked just fine in the past on older versions.

Test Repo

You can easily reproduce this bug by following the steps in my test repository.

Environment

ChristianMurphy commented 3 years ago

Some broader context, unified, remark, retext, and rehype are moving to ESM https://github.com/unifiedjs/unified/issues/121 The error you are seeing:

Only file and data URLs are supported by the default ESM loader

is raised because one or more of the plugins or presets in your configuration uses ESM, but the configuration file itself is in CJS. In your specific case this package is retext-english, which you updated from version 3 to version 4 which is ESM only https://github.com/retextjs/retext/releases/tag/8.0.0

There are two ways you can resolve this:

1. migrate to ESM config (preferred)

First rename .remarkrc.js to .remarkrc.mjs

Then update the config from CJS:

module.exports = {
  "plugins": [
    [
      require("remark-retext"),
      require("unified")()
        .use(require("retext-english")),
    ],
  ],
};

to ESM:

import remarkRetext from "remark-retext";
import unified from "unified";
import retextEnglish from "retext-english";

const config = {
  plugins: [[remarkRetext, unified().use(retextEnglish)]],
};

export default config;

Doing this will allow you to use the latest versions of remark and remark plugins.

2. Continue to use the CJS version

lock the plugins to the last major version that was CJS based in package.json. For retext-english that would be the ^3.0.0 version range. Choosing this option, will keep the existing config working, but will not get new features and fixes added in new releases of plugins.

Zamiell commented 3 years ago

Thanks for the quick reply Christian. I converted my config to the mjs format and I am good to go now.

I experienced some additional pain getting the file to lint though, which I will document here:

  1. I had to add "!.remarkrc.mjs" to the ignorePatterns section of my .eslintrc.js file, because ESLint will explicitly ignore hidden files by default. (This is necessary even if you put .remarkrc.mjs in the include section of your tsconfig.eslint.json file.)
  2. I had to add extraFileExtensions: [".mjs"], to the parserOptions section of my .eslintrc.js file, because ESLint refuses to parse mjs files otherwise.
Zamiell commented 3 years ago

Actually, I'm not all set.

Remark does not seem to be automatically finding my ".remarkrc.mjs" file. So simply renaming it from ".remarkrc.js" to ".remarkrc.mjs" as Christian describes above does not seem to be sufficient. Instead, Remark is just using no configuration at all, and reporting that there are 0 linter errors, which is why I was initially mislead into thinking that everything was working.

Furthermore, when I attempt to manually specify my config to Remark, I get an error:

$ npx remark test.md -r .remarkrc.mjs
test.md
  1:1  error  Error: Cannot parse given file `.remarkrc.mjs`
Error [ERR_UNSUPPORTED_ESM_URL_SCHEME]: Only file and data URLs are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'd:'
    at new NodeError (node:internal/errors:363:5)
    at Loader.defaultResolve [as _resolve] (node:internal/modules/esm/resolve:825:11)
    at Loader.resolve (node:internal/modules/esm/loader:89:40)
    at Loader.getModuleJob (node:internal/modules/esm/loader:242:28)
    at Loader.import (node:internal/modules/esm/loader:177:28)
    at importModuleDynamically (node:internal/modules/cjs/loader:1049:29)
    at importModuleDynamicallyWrapper (node:internal/vm/module:437:21)
    at importModuleDynamically (node:vm:382:43)
    at exports.importModuleDynamicallyCallback (node:internal/process/esm_loader:30:14)
    at loadFromAbsolutePath (D:\Repositories\hanabi.github.io\node_modules\unified-engine\lib\configuration.js:305:5)
    at FormattedError (D:\Repositories\hanabi.github.io\node_modules\fault\index.js:29:12)
    at onparse (D:\Repositories\hanabi.github.io\node_modules\unified-engine\lib\find-up.js:81:11)
    at done (D:\Repositories\hanabi.github.io\node_modules\trough\wrap.js:55:16)

× 1 error

Some questions:

1) Why isn't Remark automatically picking up my ".remarkrc.mjs" file? 2) Is it possible to manually specify the location of a ".remarkrc.mjs" file? 3) Where is the MJS file format documented? 4) Is there a --verbose flag or something along these lines so that I can get Remark to tell me the file path of the configuration that it is using?

ChristianMurphy commented 3 years ago
  1. There a number of possible causes, start by running in debug mode https://github.com/unifiedjs/unified-args#debugging
  2. https://github.com/unifiedjs/unified-args#--config
  3. https://github.com/unifiedjs/unified-engine/blob/main/doc/configure.md
  4. Again debug mode may help https://github.com/unifiedjs/unified-args#debugging
Zamiell commented 3 years ago
$ DEBUG="*" npx remark test.md
2021-07-23T21:47:33.931Z unified-engine:configuration Looking for `[ '.remarkrc', '.remarkrc.js', '.remarkrc.yml', '.remarkrc.yaml' ]` configuration files
2021-07-23T21:47:33.932Z unified-engine:configuration Looking for `remarkConfig` fields in `package.json` files
2021-07-23T21:47:33.936Z unified-engine:find-up No files found for `D:\Repositories\hanabi.github.io\test.md`

In the first "Looking for" line, it doesn't mention that it is looking for a ".remarkrc.mjs" file. Does that imply a bug in Remark?

For reference:

$ npx remark --version
remark: 13.0.0, remark-cli: 9.0.0
Zamiell commented 3 years ago

More info: Even when I manually specify the config file, it says "no files found" for some reason, which also seems to be a bug:

$ DEBUG="*" npx remark misc/FAQ.md --config .remarkrc.mjs
2021-07-23T22:13:38.020Z unified-engine:configuration Looking for `[ '.remarkrc', '.remarkrc.js', '.remarkrc.yml', '.remarkrc.yaml' ]` configuration files
2021-07-23T22:13:38.021Z unified-engine:configuration Looking for `remarkConfig` fields in `package.json` files
2021-07-23T22:13:38.025Z unified-engine:find-up No files found for `D:\Repositories\hanabi.github.io\.remarkrc.mjs`
2021-07-23T22:13:38.026Z unified-engine:file-set-pipeline:stdin Ignoring `streamIn`
2021-07-23T22:13:38.026Z unified-engine:file-pipeline:read Reading `D:\Repositories\hanabi.github.io\.remarkrc.mjs` in `utf8`
2021-07-23T22:13:38.026Z unified-engine:file-pipeline:read Reading `D:\Repositories\hanabi.github.io\misc\FAQ.md` in `utf8`
2021-07-23T22:13:38.027Z unified-engine:file-pipeline:read Read `D:\Repositories\hanabi.github.io\.remarkrc.mjs` (error: null)
2021-07-23T22:13:38.027Z unified-engine:file-pipeline:read Read `D:\Repositories\hanabi.github.io\misc\FAQ.md` (error: null)
2021-07-23T22:13:38.028Z unified-engine:find-up No files found for `D:\Repositories\hanabi.github.io\.remarkrc.mjs`
2021-07-23T22:13:38.028Z unified-engine:file-pipeline:configure Using settings `{}`
2021-07-23T22:13:38.028Z unified-engine:file-pipeline:configure Using `0` plugins
2021-07-23T22:13:38.028Z unified-engine:file-pipeline:parse Parsing `misc\FAQ.md`
2021-07-23T22:13:38.064Z unified-engine:file-pipeline:parse Parsed document
2021-07-23T22:13:38.064Z unified-engine:file-pipeline:transform Transforming document `misc\FAQ.md`
2021-07-23T22:13:38.064Z unified-engine:file-pipeline:transform Transformed document (error: null)
2021-07-23T22:13:38.064Z unified-engine:file-pipeline:queue Queueing `misc\FAQ.md`
2021-07-23T22:13:38.064Z unified-engine:file-pipeline:queue Interupting flush: `.remarkrc.mjs` is not finished
2021-07-23T22:13:38.064Z unified-engine:file-pipeline:queue `misc\FAQ.md` can be flushed
2021-07-23T22:13:38.064Z unified-engine:file-pipeline:queue Not flushing: some files cannot be flushed
2021-07-23T22:13:38.064Z unified-engine:file-pipeline:configure Using settings `{}`
2021-07-23T22:13:38.064Z unified-engine:file-pipeline:configure Using `0` plugins
2021-07-23T22:13:38.065Z unified-engine:file-pipeline:parse Parsing `.remarkrc.mjs`
2021-07-23T22:13:38.068Z unified-engine:file-pipeline:parse Parsed document
2021-07-23T22:13:38.068Z unified-engine:file-pipeline:transform Transforming document `.remarkrc.mjs`
2021-07-23T22:13:38.068Z unified-engine:file-pipeline:transform Transformed document (error: null)
2021-07-23T22:13:38.068Z unified-engine:file-pipeline:queue Queueing `.remarkrc.mjs`
2021-07-23T22:13:38.068Z unified-engine:file-pipeline:queue `.remarkrc.mjs` can be flushed
2021-07-23T22:13:38.068Z unified-engine:file-pipeline:queue `misc\FAQ.md` can be flushed
2021-07-23T22:13:38.068Z unified-engine:file-pipeline:queue Flushing: all files can be flushed
2021-07-23T22:13:38.069Z unified-engine:file-pipeline:stringify Not compiling document without output settings
2021-07-23T22:13:38.069Z unified-engine:file-pipeline:copy Not copying
2021-07-23T22:13:38.069Z unified-engine:file-pipeline:stdout Ignoring writing to `streamOut`
2021-07-23T22:13:38.069Z unified-engine:file-pipeline:file-system Ignoring writing to file-system
2021-07-23T22:13:38.069Z unified-engine:file-pipeline:stringify Not compiling document without output settings
2021-07-23T22:13:38.069Z unified-engine:file-pipeline:copy Not copying
2021-07-23T22:13:38.069Z unified-engine:file-pipeline:stdout Ignoring writing to `streamOut`
2021-07-23T22:13:38.069Z unified-engine:file-pipeline:file-system Ignoring writing to file-system
.remarkrc.mjs: no issues found
misc\FAQ.md: no issues found

(I've manually set up the FAQ.md file to include errors that Remark should catch.)

ChristianMurphy commented 3 years ago

In the first "Looking for" line, it doesn't mention that it is looking for a ".remarkrc.mjs" file. Does that imply a bug in Remark?

Most likely you locked yourself on an older version of unified engine (a transitive dependency brought in by remark), version 8.2.0 or higher of unified engine is needed https://github.com/unifiedjs/unified-engine/releases/tag/8.2.0

Try running npm upgrade or clearing your lock file and installing again.

Zamiell commented 3 years ago

Hey Chris,

Thanks for the help. I've deleted my package-json file and nuked my node_modules directory. But after doing an npm install, the same problem still persists. Is there a flag in remark-cli to print out what version of the unified-engine that the remark-cli is using?

Below, I run a find command to print out the version of the unified-engine that I have installed:

$ find . -wholename "*unified-engine/package.json" | xargs grep version
  "version": "8.2.0",

This matches the latest version that you describe in your last post. Here's unified itself:

$ find . -wholename "*unified/package.json" | xargs grep version
./node_modules/remark-admonitions/node_modules/unified/package.json:  "version": "8.4.2",
./node_modules/retext-diacritics/node_modules/unified/package.json:  "version": "10.0.1",
./node_modules/retext-english/node_modules/unified/package.json:  "version": "10.0.1",
./node_modules/retext-indefinite-article/node_modules/unified/package.json:  "version": "10.0.1",
./node_modules/retext-redundant-acronyms/node_modules/unified/package.json:  "version": "10.0.1",
./node_modules/retext-repeated-words/node_modules/unified/package.json:  "version": "10.0.1",
./node_modules/retext-sentence-spacing/node_modules/unified/package.json:  "version": "10.0.1",
./node_modules/retext-syntax-mentions/node_modules/unified/package.json:  "version": "10.0.1",
./node_modules/retext-syntax-urls/node_modules/unified/package.json:  "version": "10.0.1",
./node_modules/unified/package.json:  "version": "9.2.0",

Not sure what else to try. Do you see anything else wrong? Or is this a bug in remark itself?

wooorm commented 3 years ago

@Zamiell Could you update your reproduction repo with the error you’re seeing now so I can ivnestigate?

Zamiell commented 3 years ago

@wooorm I have updated the repo. Here are the new steps to reproduce:

  1. git clone https://github.com/Zamiell/remark-test.git
  2. cd remark-test
  3. npm ci
  4. DEBUG="*" npx remark README.md

The first line of the output will be as follows:

2021-07-24T16:29:13.746Z unified-engine:configuration Looking for `[ '.remarkrc', '.remarkrc.js', '.remarkrc.yml', '.remarkrc.yaml' ]` configuration files

This appears to be a bug, because there should be an entry in the array with a string of ".remarkrc.mjs", but there isn't.

Also, can you reopen this issue?

wooorm commented 3 years ago

So running this by adding the config file explicitly does work for me:

``` $ DEBUG="*" npx remark README.md --rc-path .remarkrc.mjs unified-engine:configuration Looking for `[ '.remarkrc', '.remarkrc.js', '.remarkrc.yml', '.remarkrc.yaml' ]` configuration files +0ms unified-engine:configuration Looking for `remarkConfig` fields in `package.json` files +2ms unified-engine:find-up No files found for `/Users/tilde/Projects/oss/remark-test/README.md` +0ms unified-engine:file-set-pipeline:stdin Ignoring `streamIn` +0ms unified-engine:file-pipeline:read Reading `/Users/tilde/Projects/oss/remark-test/README.md` in `utf8` +0ms unified-engine:file-pipeline:read Read `/Users/tilde/Projects/oss/remark-test/README.md` (error: null) +1ms unified-engine:find-up Checking given file `/Users/tilde/Projects/oss/remark-test/.remarkrc.mjs` +2ms unified-engine:find-up Read given file `/Users/tilde/Projects/oss/remark-test/.remarkrc.mjs` +40ms unified-engine:file-pipeline:configure Using settings `{}` +0ms unified-engine:file-pipeline:configure Using `1` plugins +0ms unified-engine:file-pipeline:configure Using plugin `remark2retext`, with options `undefined` +0ms unified-engine:file-pipeline:parse Parsing `README.md` +0ms unified-engine:file-pipeline:parse Parsed document +8ms unified-engine:file-pipeline:transform Transforming document `README.md` +0ms unified-engine:file-pipeline:transform Transformed document (error: null) +5ms unified-engine:file-pipeline:queue Queueing `README.md` +0ms unified-engine:file-pipeline:queue `README.md` can be flushed +1ms unified-engine:file-pipeline:queue Flushing: all files can be flushed +0ms unified-engine:file-pipeline:stringify Compiling `README.md` +0ms unified-engine:file-pipeline:stringify Serialized document +3ms unified-engine:file-pipeline:copy Not copying +0ms unified-engine:file-pipeline:stdout Writing document to `streamOut` +0ms # remark-test This is a test repository that demonstrates a bug with the latest version.
## Steps to Reproduce 1. `git clone https://github.com/Zamiell/remark-test.git` 2. `cd remark-test` 3. `npm ci` 4. `DEBUG="*" npx remark README.md` The first line of the output will be as follows: 2021-07-24T16:29:13.746Z unified-engine:configuration Looking for `[ '.remarkrc', '.remarkrc.js', '.remarkrc.yml', '.remarkrc.yaml' ]` configuration files This appears to be a bug, because there should be an entry in the array with a string of ".remarkrc.mjs", but there isn't. unified-engine:file-pipeline:file-system Ignoring writing to file-system +0ms README.md: no issues found ```

I believe you were seeing an issue with that? Or not?

This appears to be a bug, because there should be an entry in the array with a string of ".remarkrc.mjs", but there isn't.

We could add it. It’s more of a feature though. The more files we add, the slower it gets for everyone.

Zamiell commented 3 years ago

I ran the same command, and get different results:

$ DEBUG="*" npx remark README.md --rc-path .remarkrc.mjs
2021-07-24T17:27:01.383Z unified-engine:configuration Looking for `[ '.remarkrc', '.remarkrc.js', '.remarkrc.yml', '.remarkrc.yaml' ]` configuration files
2021-07-24T17:27:01.385Z unified-engine:configuration Looking for `remarkConfig` fields in `package.json` files
2021-07-24T17:27:01.388Z unified-engine:find-up No files found for `D:\Repositories\remark-test\README.md`
2021-07-24T17:27:01.389Z unified-engine:file-set-pipeline:stdin Ignoring `streamIn`
2021-07-24T17:27:01.390Z unified-engine:file-pipeline:read Reading `D:\Repositories\remark-test\README.md` in `utf8`
2021-07-24T17:27:01.390Z unified-engine:file-pipeline:read Read `D:\Repositories\remark-test\README.md` (error: null)
2021-07-24T17:27:01.390Z unified-engine:find-up Checking given file `D:\Repositories\remark-test\.remarkrc.mjs`
2021-07-24T17:27:01.392Z unified-engine:find-up Only file and data URLs are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'd:'
2021-07-24T17:27:01.392Z unified-engine:file-pipeline:queue Queueing `README.md`
2021-07-24T17:27:01.392Z unified-engine:file-pipeline:queue Flushing: all files can be flushed
2021-07-24T17:27:01.392Z unified-engine:file-pipeline:stringify Not compiling failed document
2021-07-24T17:27:01.392Z unified-engine:file-pipeline:copy Not copying
2021-07-24T17:27:01.392Z unified-engine:file-pipeline:stdout Ignoring writing to `streamOut`
2021-07-24T17:27:01.392Z unified-engine:file-pipeline:file-system Ignoring writing to file-system
README.md
  1:1  error  Error: Cannot parse given file `.remarkrc.mjs`
Error [ERR_UNSUPPORTED_ESM_URL_SCHEME]: Only file and data URLs are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'd:'
    at new NodeError (node:internal/errors:363:5)
    at Loader.defaultResolve [as _resolve] (node:internal/modules/esm/resolve:825:11)
    at Loader.resolve (node:internal/modules/esm/loader:89:40)
    at Loader.getModuleJob (node:internal/modules/esm/loader:242:28)
    at Loader.import (node:internal/modules/esm/loader:177:28)
    at importModuleDynamically (node:internal/modules/cjs/loader:1049:29)
    at importModuleDynamicallyWrapper (node:internal/vm/module:437:21)
    at importModuleDynamically (node:vm:382:43)
    at exports.importModuleDynamicallyCallback (node:internal/process/esm_loader:30:14)
    at loadFromAbsolutePath (D:\Repositories\remark-test\node_modules\unified-engine\lib\configuration.js:305:5)
    at FormattedError (D:\Repositories\remark-test\node_modules\fault\index.js:29:12)
    at onparse (D:\Repositories\remark-test\node_modules\unified-engine\lib\find-up.js:81:11)
    at done (D:\Repositories\remark-test\node_modules\trough\wrap.js:55:16)

× 1 error

Maybe it is because I am on Windows 10? I don't know.

We could add it. It’s more of a feature though. The more files we add, the slower it gets for everyone.

Perhaps I am confused, but I don't see how this is a feature. Let me ask it this way: Why is remark-cli looking for a '.remarkrc.js' file by default at all when the program crashes and burns when a '.remarkrc.js' file is loaded? That is simply a bug in the program, right?

Also, I will ask once again, can we reopen this issue?

wooorm commented 3 years ago

Maybe it is because I am on Windows 10? I don't know.

Somewhat, I think this is a Windows-specific bug. https://github.com/unifiedjs/unified-engine/commit/3857a54ef2e502dd6f0e6fdca6a497b7945b40d4 would fix it if something like it would land on the v8 branch

Why is remark-cli looking for a '.remarkrc.js' file by default at all when the program crashes and burns when a '.remarkrc.js' file is loaded?

The reason it looks for a .js file, is because JavaScript is supported. Either CommonMark or modules. However, it is only as such supported when you use CJS normally, or when you use ESM and add a type: module. It does not crash. Node forbids using static imports statements in what is otherwise a CommonJS script. More on ESM: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c. unified-engine also looks for a couple of other files, like YAML and JSON. But it’s not scalable to support every config format or every extension.

Also, I will ask once again, can we reopen this issue?

This is not an issue in this project, and is still being triaged. I’d rather open an issue in the correct place about the actual problem (which I’m trying to figure out)

Zamiell commented 3 years ago

Thanks for the quick reply woorm.

The reason it looks for a .js file, is because JavaScript is supported. Either CommonMark or modules. However, it is only as such supported when you use CJS normally, or when you use ESM and add a type: module. It does not crash. Node forbids using static imports statements in what is otherwise a CommonJS script. More on ESM: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c. unified-engine also looks for a couple of other files, like YAML and JSON. But it’s not scalable to support every config format or every extension.

The two options that Christian outlined in his original post are to: 1) switch to ESM 2) stay on old versions forever

I think the obvious choice for end users is to pick option 1. But if end-users are 1) forced to switch to ESM when using the latest version, 2) but remark no longer automatically picks up their remarkrc files

then that is a regression, right?

Maybe I should ask the question in a different way: Is there a way for end-users to: 1) use the latest version of remark 2) and simultaneously have remark automatically pick up a remarkrc file?

If so, I don't think it has been discussed in this thread thus far. I think that in general, end-users expect a tool to automatically pick up a rc file without having to manually specify it, like Bash does, like ESLint does, like Prettier does, etc.

ChristianMurphy commented 3 years ago

An alternative would be to encourage adding "type": "module" to package.json which allows ESM to be used in plain .js files. Either would be fine by me, thoughts @wooorm?

Zamiell commented 3 years ago

Any updates on this issue? Can we reopen this issue, or get an issue opened in the right place?

wooorm commented 3 years ago

Hi! Thanks for the ping and your patience.

I’ve had some time to land a fix to include .remarkrc.json, .remarkrc.cjs, and .remarkrc.mjs in the latest unified-engine. Back when this issue was opened, remark-cli was not yet released with that version of the engine, and the ecosystem was in flux. Now things have stabilized, and you can update remark-cli (or anything) to the latest version for them to work. Thus, updating remark-cli in your test repo, will result in the lines you wanted:

  unified-engine:configuration Looking for `[
  '.remarkrc',
  '.remarkrc.json',
  '.remarkrc.cjs',
  '.remarkrc.mjs',
  '.remarkrc.js',
  '.remarkrc.yaml',
  '.remarkrc.yml'
]` configuration files +0ms

Assuming that you also install unified, use its latest version (which was not a dependency in your package.json yet), and change:

-import unified from "unified";
+import {unified} from "unified";

...then everything works.

It’s been a bit of a long thread and I’m not sure if there are other things you wanted to have differently?

Zamiell commented 3 years ago

Yes, this issue appears to be resolved now in the latest versions of everything, thanks again wooorm.