Command-line parameter to force "module" instead of "commonjs" type?

[getify]

I'm aware that I can name a file with .mjs or set "type": "module" in package.json. I'm also aware of the --input-type=module CLI parameter.

What I'm not understanding is, why isn't there a way to pass a parameter flag to force module interpretation for the .js file I specify? Was there a reason that --input-type can only be used with string input and not to control module interpretation of a .js file?

I tried searching old issues to find this discussed but my searching failed me. I feel certain it must have been intentionally omitted, but I'm just trying to understand why?

[benjamingr]

@nodejs/modules

[benjamingr]

I vaguely recall a --entry-type

[benjamingr]

Further discussion from the modules repo https://github.com/nodejs/modules/issues/296

I found two more issues, I think it's better to wait for someone who was active at the modules team to say what ended up happening that caused --entry-type (or --x-types) to be removed.

Edit: found it https://github.com/nodejs/node/pull/27184

Edit2: found the reasoning https://github.com/nodejs/modules/issues/300#issuecomment-477728981

[getify]

Thanks for helping me find those links.

I completely disagree with the reasoning stated there (I don't care if it was "package-type" or "entry-type", just would want some way to do it)... but I don't have the energy to try to re-litigate it.

[DerekNonGeneric]

@getify, those decisions aren't etched in stone. We may be able to accommodate you especially if you would be interested in making a pull request. Having a flag as you describe does seem useful to me.

[ljharb]

If you're writing a file that has no associated package.json, how would you expect to convey (for invocations beyond the current one) the parse goal of the file beyond the extension, given that .js means Script and .mjs means Module, by default?

[targos]

Would there be a problem if we added a flag that allows the user to change the default behaviour ?

[ljharb]

@targos it would encourage people to use a file extension that doesn’t match the parse goal of their file, without an accompanying package.json to carry that information.

[WebReflection]

one more time: this is not a module

import {random} from 'library';
console.log(random());

the fact .mjs means module is absolutely misleading for any program that is not a module, just a program, written in JavaScript (.js).

All main bundles/files on the Web also are not modules: these are programs, that use a module system, yes, but these are not modules.

index.mjs is not a module, if it doesn't export a thing, is a program.

But yeah, this has been discussed for ages already.

[WebReflection]

Last one, for correctness, and completeness, sake:

<script type="module" src="program.js">

The type there doesn't practically define the parsing goal, it defines the module system, and it propagates with it, because program.js will import files, and these files implicitly inherit the module system, hence the parsing goal.

Something like:

node --type=module --force-type=true program.js

could enforce the module system and its propagation, so that by default everything imported by program.js would have a module system, by default, that is ESM, not CommonJS.

That's how you could define the parsing goal, but there's never been interest in doing this, although tons of developers keep asking for this, so I wish this common request was more welcomed, or developers expectations acknowledged.

I don't have the energy to try to re-litigate it

'cause this is sad, but from personal experience, also true ... current state is more opinionated than concrete, imho, while competitors factor out this gotcha for developers (see deno).

edit

node --default-type=module program.js
node --input-type=module --override-default-input-type=true

all variants that could make it, if there was the will to do so.

[getify]

Background

I built a tool called moduloze that converts a tree of commonjs-style files to either UMD or ESM style files. It's designed either to be used as a one-time codemod for code authors wishing to permanently convert to ESM, or (my use) for package authors who still prefer to write in commonjs but who want to distribute also in UMD and ESM formats for wider consumption flexibility. I have all my main/active npm packages using it.

Along with that, I also have a tool called import-remap which allows you to apply import-map style rewriting of your import specifiers as a build-tool (instead of runtime aliasing). I also use import-remap in some of my projects to modify the output from moduloze to be ESM that's more readily useable in the browser.

So, taken together, these two tools give authors a narrower-focused, and more flexible/less opinionated path (than, say, typical all-in-one bundlers) for supporting code that's CJS, UMD, and ESM, at the discretion of the users of my tools.

Note: It is not a goal to interop between formats, but to support building parallel trees of code for each format.

In that spirit, these tools (moduloze particularly) allow you to specify that you want your built ESM modules to be either .mjs or .js file extension. I offer that choice because I don't want my opinions on that topic to limit others. If they choose to make .js files with ESM code in them, they are responsible to either use that code in an environment like the browser where the script tag conveys the format, or in Node with some supporting configuration (e.g., a package.json).

Use-Case

I would like to be able to test the various combinations of outputs from my tools (both automated test suites, and user-opt-in verification steps when the tools are used on user-code).

I find it annoying/inconvenient to have to create a stub package.json file in a directory of built ESM-containing .js files with nothing but "type": "module" in it, just to force Node to treat this set of files with the proper parsing format.

It's mildly annoying to need it for my tool test suite, but it's especially annoying for the opt-in at-use-time verifications performed on user-code the tool just outputted, since such a file has to be created just to invoke Node then deleted right after... every time.

I would strongly prefer to have a flag like --package-type for that purpose.

---

I've read all the linked threads (and comments here) and I'm aware of the objections to such a feature -- i.e., general users can't/shouldn't be trusted with such a feature.

I disagree with these assertions, but don't need to have them repeated here, and I don't care to re-debate it. I've stated my use-case for posterity sake. I'll leave it at that, as I suspect it won't sway the strong opinions held by some in charge of Node's direction with modules.

[WebReflection]

FWIW, --package-type=... is more semantic than any flag I've previously written/mentioned:

• it's semantic, it mimics the {"type": "..."} in package.json
• it works for both ESM and CJS
• it overwrites the need to look for a type in the package, and it doesn't need a fallback
• it's an explicit developer intent to disambiguate the content of files in a folder (excluding explicit types as .cjs or .mjs)
• it makes the whole "we can't disamiguate ..." or "that makes developers do things we don't want them to do ..." arguments irrelevant, as it promotes developers expectations over "ambiguity/patronizing"

It'd be great if this issue would be re-considered/opened (among others) instead of keep hearing, here and there, alternatives are much better, so thanks in advance for considering this.

[aduh95]

II think the discussion would be more productive if there were a PR implementing such a flag. Let's reopen to see if there are volunteers to work on that.

[benjamingr]

FWIW I think the ask here is pretty reasonable. @ljharb given the use case and the suggestion (a --package-type flag that exactly acts like a "type": "module" in the package.json WDYT?

[ljharb]

@benjamingr --package-type was what we originally had; it was removed because a modules group member/node collaborator felt it was a footgun.

Personally, I think "type": "module" was a mistake at all, and I'd prefer not adding more ways to complicate the already very confusing landscape of parse goals and file extensions.

[WebReflection]

@aduh95 thanks for re-opening, I do appreciate that. A quick comment about your thoughts:

I think the discussion would be more productive if there were a PR implementing such a flag

I maintain tons of OS packages, and I usually ask developers landing PRs without previous conversation, or even issues to mention in such PR, to write an issue first, so we can discuss and decide what to do.

Landing a PR is very time consuming, surely more time consuming than filing an issue to understand the following:

• maybe I didn't realize it was previously discussed
• maybe somebody is already working on it, so my PR might be a duplicate
• maybe there's no interest in enabling such feature
• maybe some maintainer has strong personal opinions about this specific issue, so that my effort, as contributor, would be trashed right away

In the recent survey for next 10 years of NodeJS success, these concerns where asked, hence I believe known, but these kind of replies are the reason I wouldn't personally land a PR in here:

it would encourage people to use a file extension that doesn’t match the parse goal of their file, without an accompanying package.json to carry that information.

I am sure @ljharb has best intention in disambiguating or whatever, but the whole industry is assuming the default extension for JavaScript is .js, and this is mostly a NodeJS only limitation, due it's different default in core that few don't want to make optional, overridden, or explicitly disambiguated.

Most CLI software I use, have features and defaults, and flags to change these features or default ... and node offers tons of flag, but it's been a constant push back to offer a flag most developers expect to have:

• TS transpiles to JS files with ESM, 'cause that's the standard
• gjs uses -m flag to enable ESM module system (they had their proprietary module system too)
• jsc uses -m flag to enable ESM module system too
• deno didn't even bothered with this debate and adopted the de-facto, and official, standard

Other engines allow to disambiguate the entry point module system, and propagate it from there on.

SQLite3 offers a .mode option too, and other CLI interfaces have a way to permanently store/use the same, desired, default.

Enough counter-examples though, my whole point is: if a PR lands, are we going to push it back for the sake of it, or there is a better way to reach consensus on something highly demanded out there in 2021?

Thanks for any kind of outcome, I might land the PR myself if I understand this is something really worth my PR time.

Last, but not least, have a nice rest of the Sunday 👋

[ljharb]

It's an "anything that's not a browser" limitation. In every other part of software, file extensions are how you determine the parse goal for textual formats. That some of these systems may have added a flag doesn't mean they had to.

[benjamingr]

it was removed because a modules group member/node collaborator felt it was a footgun.

I'll be explicit: who is currently blocking --package-type? I'd like to hear them and their take on why it's a foot-gun at the moment.

[benjamingr]

@WebReflection , I am happy to contribute --package-type as I'm sure are a few others, it's not a lot of work and it's not blocked on a PR it's blocked on discussion of whether or not it's a good idea. If a collaborator is blocking this (or really, if anyone, collaborator or not is objecting to this) our process is to hear them out and reach consensus.

Given how aggressive discussion can unfortunately sometimes get in areas @getify previously contributed in like promises and modules - I absolutely understand and sympathise with his reluctance to participate in a long discussion about this. All he said is "here is my use case, it would be nice if I was able to do this".

[WebReflection]

@benjamingr thanks for clarification, but all I am saying is that these answers:

It's an "anything that's not a browser" limitation. In every other part of software, file extensions are how you determine the parse goal for textual formats. That some of these systems may have added a flag doesn't mean they had to.

after I've explicitly mentioned most other not browsers dealing with .js just fine through a flag, are what make these issues, PRs, discussions, unpleasant.

There are no evidences, and no reasons, to not allow an explicit disambiguation via a flag, and yet these answers keep popping up.

[benjamingr]

There are no evidences, and no reasons, to not allow an explicit disambiguation via a flag, and yet these answers keep popping up.

Jordan said there was a modules team decision to remove --package-type, I think the logical next step is to give whomever made the compelling argument to remove it a chance to articulate why it was a bad idea and provide these "evidences" and "reasons".

I'm sorry that discussions abound modules are often filled with baggage. It's pretty unfortunate and I honestly don't know how to fix it. For what it's worth I don't think Jordan is being aggressive here (at least not on purpose) though I can see why such a statement can appear aggressive which is unfortunate.

[WebReflection]

Like I’ve said, I’m sure he has best intentions, but the web itself disambiguate with a type flag, and everyone is moving toward that parsing goal, as JS, as defined by standards, never had a module system, hence it has practically nothing to disambiguate, as extension, because that syntax was forbidden before.

disambiguating is necessary though, and a flag does a much better, and explicit job, than crawling a folder and all its parents, to find the package.json in charge of such disambiguation.

he also said that package was a mistake, but I see it as more footgun prone than an explicit flag, ‘cause when you install anything nodejs related, and you expect CJS as default, if there’s a package in the parent folder opting in for ESM, nothing works as expected anyway.

Accordingly, I’m looking forward to read evidences around this topic, compared to the current state.

thanks for facilitating this 👍

[ljharb]

@WebReflection i do totally agree that the package.json flag is much more of a footgun than a CLI flag would be.

[WebReflection]

@ljharb that's good to hear, so maybe this conversation could be moved forward into a better direction:

• if a program is run and the folder of such program doesn't disambiguate the parsing goal, a warning is shown in console right away to help developers migrate to an explicit parsing goal (by flag or by proper package.json type and exports definition)
• package.json is still the module parsing decider for bare imports through the node_modules folders
• extensions such as .cjs and .mjs have explicit priority over everything
• subfolders still benefit from the crawling dance that looks for package.json
• the whole parent-folder crawling becomes obsolete and discouraged by warnings
• ambiguity of the module system will fade away in time
• we are all happy

wouldn't discussing an improvement, and welcome a more explicit intent, be then a better way forward?

[bmeck]

We had an old flag in https://github.com/nodejs/node/pull/32394 which might be good to look over. The main concerns of that PR don't appear to be discussed here. In particular understanding how to deal w/ ahead of time tooling needing to know under which flag a file will be executed is complex and the reason it was provided as a build time option and not a run time option in that PR. I'm not exactly keen on adding configuration options that force specific CLI options to be active for programs to work, but we do have some precedent. A plan for how to handle packages expecting to have this flag set in conflicting manners would greatly alleviate my concerns; however, until such a plan is explained I'd be -1 on this run time flag approach being litigated again. With the build time flag for any given node build the meaning of a file remains unambiguous for ahead of time tooling, but with a run time flag all files that are able to change no longer have that invariant for ahead of time tooling.

[bmeck]

I'd also note that changing the default package type tends to break tooling that assumes formats; playing around with that PR and do things like figure out how to make the node core test suite not break would be a good first step.

[benjamingr]

In particular understanding how to deal w/ ahead of time tooling needing to know under which flag a file will be executed is complex and the reason it was provided as a build time option and not a run time option in that PR.

I am not arguing whether or not this is a good or bad design, but we have a lot of flags in Node.js that control that ranging a spectrum of areas like:

• Dealing with sync/async uncaught errors or stuff that becomes errors - --abort-on-uncaught-exception --unhandled-rejection=warn --throw-deprecation etc.
• Loading arbitrary code required for the program to run --require --loader.
• Controlling behaviour of various APIs - --zero-fill-buffers --preserve-symlinks and so on.

So it's not just that there is prior art for command line flags doing this - there is a lot of flags observably changing the runtime behavior of a Node.js program. Again: I am not saying it's a good idea or a bad idea just that it's hardly new.

A plan for how to handle packages expecting to have this flag set in conflicting manners would greatly alleviate my concerns;

Can you elaborate on what you mean by packages expecting to have this flag set in conflicting manners? I'm sorry if this was discussed before feel free to point me to a comment explaining what that means.

[benjamingr]

Also I just want to point out that the way everyone is describing these technical discussions as "litigation" is kind of terrifying for someone like me who doesn't have all the context and most of the info 😅

[getify]

@bmeck

... understanding how to deal w/ ahead of time tooling needing to know under which flag a file will be executed is complex...

Ahead of time tooling doesn't have to make these complex choices, if it's left as a choice/option to the end-user (and they're informed of that fact). That's the approach my tooling takes.

But ahead of time tooling does make this sort of choice regularly, such as if it produces a package.json with "type": "module" in it. I'm still not seeing how the decision to allow this ahead of time decision to be baked into a package.json precludes the ability to propagate that decision through command line flags (or environment variables) , which can of course be baked into shell scripts or aliases.

Is there a concern that by adding this flag, the greater JS population will abandon their migration to a preferred .mjs file extension and jump on some --package-type=module bandwagon? Is there fear that a large chunk of the community actually really wants .js by default and the presence of this flag is an admission of that reality rather than waiting for them to all acquiesce?

[bmeck]

@benjamingr

Can you elaborate on what you mean by packages expecting to have this flag set in conflicting manners? I'm sorry if this was discussed before feel free to point me to a comment explaining what that means.

given:

A/a.js # expects to be CJS
B/b.js # expects to be ESM

If using node --package-type=module B/b.js just explaining how to deal w/ a.js potentially being loaded as ESM if we introduce this flag / how to prevent such situations so that the corollary node --package-type=common A/a.js loading b.js incorrectly also is covered in the explanation.

So it's not just that there is prior art for command line flags doing this - there is a lot of flags observably changing the runtime behavior of a Node.js program. Again: I am not saying it's a good idea or a bad idea just that it's hardly new.

Yep, however, I tend to think that this specific flag is much bigger in terms of potential issues than things like --zero-fill-buffers and the like. Most APIs can always throw due to OOM, etc. so adding throw behavior isn't really invalidating some kind of invariant when calling APIs. Expecting stale memory in your Buffer likewise isn't really a behavior that is useful for general usage, it rather expects that a program won't misuse the data and that no adversarial code is going to be run.

I'm not really here to be convinced of having 1 flag altering behavior meaning that we shouldn't consider this flag in a different light. I do think this flag being discussed is more error prone than the ones mentioned above except for --perserve-symlinks; which fortunately can't be mixed in a single application, but the concern for me is situations like A and B is above is exactly mixed expectations within a single process.

I do know of historical issues with things like child_process being invoked with/without it and causing errors (generally not with the other flags to my knowledge) and explanations of why adding that configuration complexity is worth the footguns is my main thing I need to be convinced of. I even think a build flag is fine and wrote a PR for it so I wouldn't try to generalize my stance as those of all those here but I do have a strong stance after that PR that while we can ship a flag, shipping it likely isn't worth the footguns if it introduces too much runtime variance.

[bmeck]

Ahead of time tooling doesn't have to make these complex choices, if it's left as a choice/option to the end-user (and they're informed of that fact). That's the approach my tooling takes.

I'm not sure I understand this. Leaving it up to the end-user seems to be the entire issue since it means it can accidentally be mis-used by default. Even if one section of a program is configured properly, it could be configured improperly when used within another section of the program. This stance seems to rely on the entry point being the only part that is configured.

But ahead of time tooling does make this sort of choice regularly, such as if it produces a package.json with "type": "module" in it. I'm still not seeing how the decision to allow this ahead of time decision to be baked into a package.json precludes the ability to propagate that decision through command line flags (or environment variables) , which can of course be baked into shell scripts or aliases.

I'd agree that for runtime tooling that has complete control over code generation this is not a problem. Per shell scripts and aliases, I think explanation for and documentation on how it would be required for users to utilize this feature would be sufficient and help to weigh the cost of such a flag.

Is there a concern that by adding this flag, the greater JS population will abandon their migration to a preferred .mjs file extension and jump on some --package-type=module bandwagon?

I have no preference on how a format is decided, just that it is statically known and doesn't introduce conflicts across usages.

Is there fear that a large chunk of the community actually really wants .js by default and the presence of this flag is an admission of that reality rather than waiting for them to all acquiesce?

I do not have such a fear, no. I also do find this phrasing a bit biased and confrontational and won't be likely to respond well to such tones in a constructive manner. This phrasing seems to be attempting to make an emotional response so I'd prefer we focus on more constructive discussion.

[WebReflection]

@bmeck

given:

A/a.js # expects to be CJS
B/b.js # expects to be ESM

If using node --package-type=module B/b.js just explaining how to deal w/ a.js potentially being loaded as ESM if we introduce this flag

This example is the same today, unless explicit extensions are used, so A/a.cjs and B/b.mjs is used, but this flag intent is a way for developers to guarantee their intent in loading any file, starting from the program passed as node argument, guarantee from that time on the default parsing goal is either module or commonjs ... and if such program loads files from another directory, the package.json crawling dance would apply, as it is now, nothing different.

Accordingly, what is the difference from your example, and one that use explicit extensions or explicit {"type": ...} in each package.json per folder?

The (common) use case, presented here, is that a developer has a folder, owns the folder and its subfolders, and want such folder to run as ESM/CJS by default, and if some file is not meant to be run as such, it can be explicit via extension, or package.json disambiguation, right?

Also, is there are real-world use case for the scenario you are describing? 'cause it seems ambiguous by intent, but we have everything we need these days to make such program less ambiguous.

Thanks.

[bmeck]

This example is the same today, unless explicit extensions are used, so A/a.cjs and B/b.mjs is used, but this flag intent is a way for developers to guarantee their intent in loading any file, starting from the program passed as node argument, guarantee from that time on the default parsing goal is either module or commonjs ... and if such program loads files from another directory, the package.json crawling dance would apply, as it is now, nothing different.

This is currently not ambiguous for Node programs even using .js without a package.json I don't understand the comment. The default interpretation of .js is static.

Accordingly, what is the difference from your example, and one that use explicit extensions or explicit {"type": ...} in each package.json per folder?

Such a field would be statically knowable, and the implication of the field would be it needs to be used in my example above to prevent the conflict of running a file in the wrong format.

The (common) use case, presented here, is that a developer has a folder, owns the folder and its subfolders, and want such folder to run as ESM/CJS by default, and if some file is not meant to be run as such, it can be explicit via extension, or package.json disambiguation, right?

Currently, they have those options. The discussion here is about providing a different means to do so and what that would look like.

Also, is there are real-world use case for the scenario you are describing? 'cause it seems ambiguous by intent, but we have everything we need these days to make such program less ambiguous.

I mean, this happened to a lot of tools needing to integrate against a statically knowable package.json#type field loading things in the wrong format. The conversation I'm waiting on is explanation on how a non-statically knowable configuration is going to handle similar issues. For the statically known "type" field, tools can just look at that field to know what a file is intended to be. For a CLI argument as described above, to my knowledge it is not necessarily the author of the file that controls how files are interpreted and that is a primary source of the issue.

[getify]

I also do find this phrasing a bit biased and confrontational and won't be likely to respond well to such tones in a constructive manner. This phrasing seems to be attempting to make an emotional response

It was blatant sarcasm, and I didn't make an attempt to hide that.

I deliberately didn't ascribe the claim to you personally, because it wasn't meant as an attack or emotional provocation. I'm sorry I came off that way.

The intent of the sarcasm is to point out what seems (to me) a bit of irrational resistance (not here but in the past threads) -- but wrapped up in a rational sounding argument.

This is a completely optional feature that only those motivated to use would affect. Its potential side effects if people used it wrongly, such as specifying it while running files not designed for it (like the Node test suite) is, IMO, not particularly salient.

These kinds of technical discussions almost always seem to get stuck on the pathologic corner cases and miss the forest for the absurdity.

In any case, my irritation at how often that happens is why I closed this issue earlier and said I didn't want to re-debate it. I shouldn't have spoken up.

[benjamingr]

If using node --package-type=module B/b.js just explaining how to deal w/ a.js potentially being loaded as ESM if we introduce this flag / how to prevent such situations so that the corollary node --package-type=common A/a.js loading b.js incorrectly also is covered in the explanation.

I totally see that point though I am not sure why that is different from the roundabout way of creating a package.json with a "type":"module" launching a child_process and then deleting the package.json.

That is: I see (though I'm not sure where my opinion stands) the point of the parse-goal being ambiguous being problematic - but to my understanding that is already the case today with "type": "module"?

Yep, however, I tend to think that this specific flag is much bigger in terms of potential issues than things like --zero-fill-buffers and the like.

Those were just examples of flags on top of my head - though arguably stuff like --abort-on-uncaught-exception is much more drastic regarding the observable behaviour change of programs because a flag was passed than anything else since it literally crashes the server even if a process.on('uncaughtException' listener is installed.

My point was just there was prior art for flags changing the observable difference of programs. One other obvious one on top of my head is --inspect-brk which pauses the program.

I'm not really here to be convinced of having 1 flag altering behavior meaning that we shouldn't consider this flag in a different light.

I am not mentally in the decision making phase yet and honestly don't understand the problem well enough yet to hold a strong opinion. To be clear I am not saying we should have more flags that alter runtime behaviour (only there are only a bunch) nor am I saying we should have --package-type only that I understand and sympathise with Kyle's use case.

, but the concern for me is situations like A and B is above is exactly mixed expectations within a single process.

That is a legitimate concern though typically I thought behaviour altering flags (like --unhandled-rejections or --abort-on-uncaught-exceptions) were only expected to be used by "end users" and packages did not expect to be called. I would assume a theoretical flag would behave similarly?

[bmeck]

I totally see that point though I am not sure why that is different from the roundabout way of creating a package.json with a "type":"module" launching a child_process and then deleting the package.json.

This assumes a variety of things, such as a mutable fs, access to child_process, etc. In general any static analysis is only valid for a given state and not across all possible mutations of the state in the future. Stating that if the state changes it invalidated the assumptions of a previous state doesn't need to be argued, we agree there. The claim that because a state can be mutated (such as deleting a package.json) just means that you are analyzing against a completely different state, not that the state was ambiguous in the pre or post mutation states.

That is: I see (though I'm not sure where my opinion stands) the point of the parse-goal being ambiguous being problematic - but to my understanding that is already the case today with "type": "module"?

This likely could be argued on various levels; in general my concern is about being analyzable, not that the state which resulted in a conclusion being unable to be invalidated. With the discussion of the specific flag above, I believe the state would not be analyzable in any state. The ability to analyze is still possible against an executable with a build time flag, which is why I preferred it in my PR. Shipping a 2nd executable to default to ESM would likely be preferable to me than the discussion of the flag above at this time. That would effectively change the issue from runtime configuration being ambiguous to unambiguous based upon which executable is used.

That is a legitimate concern though typically I thought behaviour altering flags (like --unhandled-rejections or --abort-on-uncaught-exceptions) were only expected to be used by "end users" and packages did not expect to be called. I would assume a theoretical flag would behave similarly?

As I explained above, the behavior of such a flag is quite different than --abort-on-uncaught-exceptions, halting progress doesn't have the same implications as potentially running incorrectly.

I do believe the general expectation would be that only an entry point needs to be configured and not any sort of dependency. We already have a few ways to configure an entry point and I'm just trying to iron out the feature being looked at.

[WebReflection]

Shipping a 2nd executable to default to ESM would likely be preferable to me than the discussion of the flag above at this time.

What is the concrete difference between an explicit flag and an explicit executable? To me an executable can be:

#!/usr/bin/env node --package-type=module

which doesn't work on Linux, but there are workarounds already and that can be any executable, 'cause executable don't care about mime type.

Accordingly, I don't see tools benefits with either cases, as tools can also statically analyze, or be instrumented, to consider a flag passed by default, isn't it?

The default interpretation of .js is static.

Imagine a new developer enters the JS landscape ... so .js in the file system means either text/javascript or application/javascript.

The developer looks up at specifications of the language, and it turns out, the default expected module system in such specification is ESM.

Search in MDN, end up in ECMAScript Specifications, and all this new developer finds is ESM.

Now, NodeJS decided that .js doesn't mean ESM, the official module system for .js, but it's CJS, while every other JS environment allows a flag to enable such module system and call it a day, without any disambiguation issue, or with a need for disambiguation (gjs is used a lot in GNOME, but it keeps being ignored in these conversations).

So, universally, .js means JavaScript, and JavaScript means ECMAScript specifications, and these specifications have zero references to CommonJS.

Can we all agree this .js disambiguation issue is mostly a NodeJS only issue, and the browsers solved the issue anyway with a flag that is type="module" in a script tag?

And as this is the evidence, and there is no counter-evidence that any developer meaning to run a folder as ESM ever had any issue, as long as explicit extensions are used when needed, and as long as packages used can disambiguate, what are the concrete concerns over this expectations?

'cause tools have been adapting to changes the specifications did over time, and dare I say it should never be vice-versa, or we're stuck in a chicken/egg case there ... so, why are tools, utilities on top of node, used as argument against what node itself should do, given its semver convention and will to move forward over time, instead of being stuck with legacy?

[getify]

What's clear to me is there's a bias held by some here that wants to avoid a feature (so called, a "footgun") where a command like node .. somefile might fail because Node didn't properly understand the intended context for interpreting that command.

I don't understand or agree with this bias, because I don't understand why Node needs to exert such control over how it's run? Why isn't it fine for Node to just fail to run a file, and that be a PEBKAC sort of situation, resolvable by consulting documentation, asking in forums, trial-and-error, etc -- and eventually fixing one's mistakes?

Why do we have to contort behaviors (in this case, withhold features) to avoid basic mistakes where someone tries to run a file a certain (invalid) way, and it fails? Can't they just figure out what they did wrong and run it differently? There's a million different ways you can configure and use Node where things will break. This contemplated flag is not, from what I can tell, in any way unique or more susceptible than all those other possible mistakes.

It's not even like these would be silent or hard-to-spot errors. The whole thing would blow up right at the time of invocation. "Boom, you passed the wrong flag, sorry, that failed. Try again."

I don't think it's Node's responsibility to always perfectly and magically understand how to run some file. It's the person (or tool, configured by a person) invoking the node executable that owns that responsibility. They have to make sure the right params, environment, dependencies, etc are all in place, so that executing a file will work as expected. They rely on good documentation to let them know what's required. As far as I can tell, that's perfectly acceptable status quo for software tools of this sort.

[ljharb]

Why would it ever be preferable to create a PEBKAC situation where one could be avoided? "Can't they just figure it out" - no, people often can not "just" figure things out. Figuring things out is hard. Good software makes the right choices easy, and the wrong choices hard, while providing good error messages to suggest the right choices.

Thus, every tool owns that responsibility - if it's used wrong, it's not the fault of the user. You don't have to agree with that philosophy, and every node collaborator may or may not, but it's the one I use for my software.

[WebReflection]

I think I agree with @getify one more time, and that's the "patronizing" part I've previously mentioned.

CLI tools have flags, defaults, and flags to override defaults ... it's a developer duty to understand flags, or defaults, and related behaviors, not the program itself to be magic in all cases ... fail fast here is key too, and magic software usually ends up giving less control, not more, and magic also usually brings more unpredictable behaviors, instead of more intended one (see current parent folder and package.json state).

[getify]

Good software makes the right choices easy, and the wrong choices hard

Sure, which is why nobody in this thread is arguing for a change of default behavior. You make it opt-in by requiring a person to add a flag to change the default behavior. Just about every tool I've ever used has default behavior and then it has flags to let you override that behavior. Some of those flags can really alter the experience of using the tool, so you may even be presented with a warning like "are you sure?".

But to suggest that the tool should not have a flag if there's any chance that someone could do something silly/wrong with it, is a micromanaging nanny-style design that I don't agree with for my tools.

---

Imagine the design discussions for the -f flag in the linux rm tool. That's a dangerous flag. You can blow away the whole file system if you're not careful. I have done it. But keeping that flag out of the tool makes the tool harder to use for power users who know what they're doing. It's not that tool designer's job to stop me from blowing away my root fs. It's my job.

[bmeck]

@WebReflection

Accordingly, I don't see tools benefits with either cases, as tools can also statically analyze, or be instrumented, to consider a flag passed by default, isn't it?

This would require the configuration apply not just to node but to every tool to know how the user is going to use node. Either by ignoring a potential of multiple options, or requiring the user to pass in their expected mode of operation. As stated for other flags above, I don't think the situation with other flags except --preserve-symlinks to really be in the same consideration for needing to be parameterized in all tooling written for node. This somewhat means in order to use the flag properly, it also needs the user to be aware of how to configure it properly in all downstream tools. Learning how to properly configure it in all the downstream tooling seems non-trivial to me still.

Imagine a new developer enters the JS landscape ... so .js in the file system means either text/javascript or application/javascript.

Generally I don't think this is the case. Most don't know the MIME and/or due to the plethora of blog posts not using ESM that still exist and continue to be written I'm not sure this is true even. Even if they are trying to use the MIME to disambiguate it, and even if they are only referring to the JS spec (which only mentions .mjs not .js, which is kind of odd), they are still not disambiguated and so might not be writing ESM. Any tutorial that covers the with statement for example likely is using .js still.

[...] these specifications have zero references to CommonJS.

Correct, we likely could add one due to interoperability considerations, but I would prefer we didn't since CommonJS isn't being standardized in ECMA262.

Can we all agree this .js disambiguation issue is mostly a NodeJS only issue, and the browsers solved the issue anyway with a flag that is type="module" in a script tag?

The browsers solved it for their concerns, yes. That doesn't mean they have the same concerns as other environments as has been stated and shown many times before.

And as this is the evidence, and there is no counter-evidence that any developer meaning to run a folder as ESM ever had any issue, as long as explicit extensions are used when needed, and as long as packages used can disambiguate, what are the concrete concerns over this expectations?

This leads back to me wanting explanations of what we want to do. Stating that I can override the behavior isn't really objected to. Explaining that we need to ensure we don't cause collisions that are problematic is what I'm seeking to see. I gave a very minimal reproduction above that isn't clear from the discussion above on how to act. One could reduce the problem even further:

/a.js # expects to run as CJS
/b.js # expects to run as ESM

The point of this feature is to be able to run things that lack a package.json and lack explicit file extensions in a manner that depends on the parameters passed at runtime. I'm seeking for this example to be explained on how to debug and fix the issue somewhat like the claims above that users will just figure it out. What are they able to figure out from running in the wrong way? What might be problems of doing so? What are the explicit benefits of lacking a package.json and not using explicit extensions that necessitate this specific situation to be configurable? etc.

'cause tools have been adapting to changes the specifications did over time, and dare I say it should never be vice-versa, or we're stuck in a chicken/egg case there ... so, why are tools, utilities on top of node, used as argument against what node itself should do, given its semver convention and will to move forward over time, instead of being stuck with legacy?

The same reason why things like Dynamic Modules were rejected by TC39, in this specific case the problem is the runtime dependent behavior changes an invariant of what can be known about the module system statically.

What's clear to me is there's a bias held by some here that wants to avoid a feature (so called, a "footgun") where a command like node .. somefile might fail because Node didn't properly understand the intended context for interpreting that command.

Correct.

I don't understand or agree with this bias, because I don't understand why Node needs to exert such control over how it's run? Why isn't it fine for Node to just fail to run a file, and that be a PEBKAC sort of situation, resolvable by consulting documentation, asking in forums, trial-and-error, etc -- and eventually fixing one's mistakes?

It doesn't necessarily assert this control, you can just use a --loader to override this. However, --loader is seen as a very advanced feature for power users. They generally invalidate most things about the module system from a static analysis perspective. From my understanding, the flag being discussed is intended for general usage. General usage that would cause too much variance in the guarantees about what a program needs in order to run properly is the concern. This variance isn't just about running the node program correctly, but also about what tools must account for general interoperability concerns with node. As a collaborator, this expands the matrix of things needed to know when running a program. Generally people don't ask for CLI flags about how to run a reproduction for a bug. Flags that invalidate things starts to make figuring out what is wrong increasingly complex and needing more and more knowledge in order to debug.

Why do we have to contort behaviors (in this case, withhold features) to avoid basic mistakes where someone tries to run a file a certain (invalid) way, and it fails? Can't they just figure out what they did wrong and run it differently? There's a million different ways you can configure and use Node where things will break. This contemplated flag is not, from what I can tell, in any way unique or more susceptible than all those other possible mistakes.

I'm not sure what this is arguing for. They have a few ways to solve the mistake already. Is the claim here that those are hard to figure out?

It's not even like these would be silent or hard-to-spot errors. The whole thing would blow up right at the time of invocation. "Boom, you passed the wrong flag, sorry, that failed. Try again."

This is less likely for entry points actually since lots of entry points don't export anything. If something just used dynamic import for example it wouldn't blow up but may behave differently.

I don't think it's Node's responsibility to always perfectly and magically understand how to run some file. It's the person (or tool, configured by a person) invoking the node executable that owns that responsibility. They have to make sure the right params, environment, dependencies, etc are all in place, so that executing a file will work as expected. They rely on good documentation to let them know what's required. As far as I can tell, that's perfectly acceptable status quo for software tools of this sort.

I'm not really arguing for perfect understanding, we already have a few power features that can invalidate pretty much any behavior in node. I do think that the person running the executable shouldn't be susceptible to increased configuration or added responsibility in the common case like this feature seems to introduce. Currently, there is no need to adopt responsibility to determine how the default behaviors of node work. This feature would effectively force users (not necessarily authors) to learn about and adopt this responsibility.

CLI tools have flags, defaults, and flags to override defaults ... it's a developer duty to understand flags, or defaults, and related behaviors, not the program itself to be magic in all cases ... fail fast here is key too, and magic software usually ends up giving less control, not more, and magic also usually brings more unpredictable behaviors, instead of more intended one (see current parent folder and package.json state).

If we could fail fast and prevent improper running, give a good story on how to avoid problems like above, and how to discover this responsibility that would greatly alleviate my concerns. So far, the main push back seems to be about a claim that the feature is not desired just because the users wanting this feature expect to have this responsibility. There isn't much conversation yet about others who would be impacted by this feature.

But to suggest that the tool should not have a flag if there's any chance that someone could do something silly/wrong with it, is a micromanaging nanny-style design that I don't agree with for my tools.

I think this brings up a good point. It might be good to figure out how the core constituencies weigh this style of design. Preferably this could be done with a clear outcome on where this feature lies on constraining vs loosening the invariants of the module system and whom that impacts and how.

Imagine the design discussions for the -f flag in the linux rm tool. That's a dangerous flag. You can blow away the whole file system if you're not careful. I have done it. But keeping that flag out of the tool makes the tool harder to use for power users who know what they're doing. It's not that tool designer's job to stop me from blowing away my root fs. It's my job.

If the point is just to have the capability, you could write a loader currently or use any of the other features mentioned above. The feature here is about having it be adopted to a common case concern in order to provide ease of use.

[getify]

From my understanding, the flag being discussed is intended for general usage.... in order to provide ease of use.

You've conflated these two things, from what I can tell, and I don't agree they're the same.

Having a feature that makes it ergonomic (aka "easy") to do something (as opposed to having to create a file, run node, then delete a file -- certainly not ergonomic) is not the same thing as creating or encouraging a general use-case pattern.

This is part of the reason for my sarcasm up thread about "fear" of this taking over... I don't think it will take over, and I don't think we should have "fear" that it might. I think there are targeted use-cases, like mine, where people know what they're doing, and want to run a .js file as a module. I don't think they should have to hack around Node to accomplish that.

The fact that a segment of the community could start using the flag more broadly, in ways you or others may not prefer, is speculative at best, and isn't a strong argument for disallowing the feature, IMO.

[bmeck]

This is part of the reason for my sarcasm up thread about "fear" of this taking over... I don't think it will take over, and I don't think we should have "fear" that it might. I think there are targeted use-cases, like mine, where people know what they're doing, and want to run a .js file as a module. I don't think they should have to hack around Node to accomplish that.

I'm still quite lost on the usage of sarcasm here. I don't think it is easy for me to understand. I didn't see the comment above as being sarcastic and I am a little confused. There isn't room for discussion it seems on using any of 3+ alternatives, 1 of which lets you avoid explicit extensions and avoid having a package.json.

The fact that a segment of the community could start using the flag more broadly, in ways you or others may not prefer, is speculative at best, and isn't a strong argument for disallowing the feature, IMO.

I think this could be stated for any given feature? Is there a specific reason this feature isn't likely to get broad adoption and/or why the concerns about figuring out what to do with it aren't valid? None of the stuff above really is claiming to block the feature, the -1 is tied to a lack of those explanations and planning.

The claim that users should need to know about various CLI arguments is concerning, but we do have precedent for it. The precedent of debugging experiences on things like --preserve-symlinks isn't something I find to be easy to most users.

We already have the capability like I stated above using a loader so the feature itself isn't really adding to the capabilities of Node, but it is moving it from a fairly power feature usability to a general usability. I'm a bit lost on this differentiation of "easy to do" and "won't be done broadly". If we have the capability as a power API usage and the feature won't be used broadly, having it easy to use should avoid general problems and not spread increased burden if used. Those things are still not really being discussed yet.

[GeoffreyBooth]

This thread has 44 comments already; perhaps we should convert it to a discussion?

To try to answer some of the early questions, @benjamingr found the best links here. In particular, https://github.com/nodejs/modules/issues/300 included a link to https://github.com/nodejs/ecmascript-modules/pull/57, which was an implementation of --package-type (albeit in a much older version of the current modules codebase). If anyone is going to attempt a PR for --package-type, I would start by reviewing those two threads. You can also find more references in https://github.com/nodejs/modules by searching for --type, which was another possible name for the flag before it settled into --entry-type / --input-type / --package-type options.

My recollection of where we left off with --package-type was not that there was strong opposition, but rather that it raised a set of tricky questions (how would tools know how to treat certain entry points, etc.) that would be hard to sort out in understandable ways. I think that was more what the “footgun” discussion was about—not that we don’t trust our users, but rather how do we design such a flag so that it behaves as users expect, and when it does error, the error makes sense (as in, it doesn’t feel like a bug) and the error message can guide the user toward the correct path without too much confusion. At the same time as we were starting to explore all these issues, the question arose of whether the flag was worth it, since perhaps just requiring the package.json key was enough; and so I think the decision was to table the flag for the time being until a user arrived with a compelling enough use case to make it clear why we also need the flag. I think that’s the decision I was summarizing in https://github.com/nodejs/modules/issues/300#issuecomment-477728981. And maybe that compelling use case has arrived, in which case sure, let’s figure out the details and open a PR; or others can make the case that we shouldn’t have such a flag for whatever reasons might not have been apparent back in 2019 before ESM shipped.

One other thing that has come up since then was that now we have ESM loaders (or rather, they’re in progress) and so a custom loader can achieve what --package-type would have been able to. So if the use case is narrow enough, like a particular build tool needs the proposed --package-type=module behavior but general users or even most tools might not, then something to consider is whether that tool should just ship a loader to achieve its needs and that could be used via --loader=./node_modules/some-tool/loader.mjs instead of --package-type=module, and if not, why not.

I would be fine with a potential --package-type flag, presuming we can work out the details and find use cases that are clear and compelling.

[getify]

Is there a specific reason this feature isn't likely to get broad adoption

Anything that's not default behavior and requires a parameter -- not a short single character one but a long one with a specific assigned value -- to opt into is just not nearly as likely to take over as the broadly common way people build and deploy node applications. I have no scientific proof for that assertion, but I think it's common sense.

The claim that users should need to know about various CLI arguments is concerning

This is probably the objection I find least compelling of all. You're concerned that people need to understand what a flag does and that its name alone isn't self-explanatory? I can conjure thousands of counter-examples, some in Node and many in similar Node-adjacent (or linux) software, where the implications of the usage of a flag require you to pay close attention and not just haphazardly throw the params on. Gzip and Git come to mind immediately.

I took a quick glance through the command-line options Node currently has (like 50 or so?) and I only understood about 10 of them by their name or short description. There's a bunch there I don't understand, and I would never dream of using them unless I took some time to read the docs.

I probably only understand 5-10% of the options available in Git, because I haven't spent the time to go learn the rest of them. But I don't resent them being there -- ostensibly somebody finds them useful.

I don't see why this parameter should be any different?

We already have the capability like I stated above using a loader

First of all, I don't know what a loader is? I'm just now hearing of it. From your implication, I'm inferring it's a programmatic extension or plugin I could write that would override Node's typical behavior of treating the .js file extension as a signal of the compilation target?

I suppose I wouldn't mind learning all about that feature -- I'm imagining it's sort of like writing a service worker for a web app -- but before I invest that time, would you indulge a few questions to clarify?

1. What manner of signals (environment variables, command line params, etc) allows me to "install" this loader so that it's used? Is it compiled into Node, or is it loaded in on a per-invocation basis?

2. How easy is it to ensure that the behavior I get from running a file through a loader works the same as if some other user had just named the file with .mjs or put the "type": "module" signal in the package.json? If it's not relatively straightforward to ensure perfect parity with those code paths, it wouldn't provide a very useful signal for my use-case (testing/verification).

3. How easy would it be for me to "ship" this loader along with the code? Does it get loaded in like other npm dependencies, or is it accessed via a different channel than node_modules?

   Half of my use-case is running test-suites for my own project, so "installing" a loader into Node to do so is not out of the question (though it is a bit intrusive). But the other half is the optional verification that someone would do in using my tool on their own code, so I would need to be able to run this loader on their system, for them. Obviously, that would be a lot easier if all I needed was to invoke their Node executable with a specific flag. If I have to inject custom plugin code into their node, this might be impractical.

[getify]

My recollection of where we left off with --package-type was not that there was strong opposition

That may be true, but I saw several comments in those threads with things like "I'm -100 on ...", which sure seems like strong objection to me. Again, that's why I was reluctant to get drawn into this discussion, because I don't have the energy to take on a 2+ year old inertia against the request.

[aduh95]

Documentation for the loader implementation in Node.js: https://nodejs.org/api/esm.html#esm_loaders

Here's a minimal reproduction (note that the package.json is completely optional, I've added it just to demonstrate the loader overrides whatever it sets):

rm -rf blank-project
mkdir blank-project
cd blank-project
echo '{"type":"commonjs"}' > package.json
echo 'export async function getFormat() { return { format: "module" } }' > loader.mjs
echo 'console.log("yay ESM");export {}' > index.js
node ./index.js || echo "Fails without the loader"
node --experimental-loader ./loader.mjs ./index.js

[getify]

@aduh95 the docs you linked to indicate that these mechanisms are being redesigned, and that the hooks may very well change or disappear. Wonder how stable these actually are, because the docs aren't terribly confidence-inspiring in terms of building tooling features on top of?

[DerekNonGeneric]

Yeah, the hooks are currently being redesigned. I really like the -m argument.