[compatability]: node + esm

currently trying to use named imports with chevrotain throws an error:

SyntaxError: The requested module 'chevrotain' is expected to be of type CommonJS, which does not support named exports. CommonJS modules can be imported by importing the default export.
For example:
import pkg from 'chevrotain';
const { createToken, Lexer } = pkg;

Node will also only see the non-esm version since it ignores the module field in the package.json. There are two fixes for this: conditional exports, nested exports (links to the docs). And in either way, there needs to be a package.json with the property "type": "module" for node to recognize it as an ESM somewhere above it in the tree. (eg. in the ts outDir)

Lack of this also effectively causes the typedefinitions to be incorrect, as they claim the package does have named exports

Additionally, setting the TS target for esm to es5 makes little sense, as there is no esm in es5. it should be set to at least es6, since that would also mean that the outputted files use those es6 features that help static analyzers.

EDIT (VITAL): node also requires file extentions to be added to import specifiers (follwing esm-spec)

Update

Took me over 3 hours, but i fixed it in my fork. I always forgot how awkward things get when i can't use the latest features. I wont be creating a PR, i just don't have the time or energy to adjust the tests etc. But feel free to ask me questions about why i did something the way i did and use my fork as a reference. It successfully builds the entire project.

Here's the clipnotes version

added file extentions to all relative import statements
added a conditional import to package.json
converted all the scripts to esm
- replaced fs-extra with builtin fs/promises (NOTE: fs-extra can't be run on node-v14)
- replaced all the readFileSync with the promised-based fs-api (NOTE: this becomes really easy with top-lvl-await we have in current node)
- adjusted other esm-stuff like replacing the now unavailable __dirpath with a proper import.meta.url reference etc.
added to the build:esm script: it now copies a dummy package.json with {"type": "module"} into esm_lib after building.
added "allowSyntheticDefaultImports": true to the ts-configs (NOTE: "regexp-to-ast" has wrong typings and ts won't be able to import it properly for usage in esm. I forgot why it worked before, (could be that it just didn't at all, but it's too much work to check right now))

why i'm not creating a PR

I just don't have the energy to adjust the tests, and i'm also not going to update/remove all the outdated devDeps. it's just too much work, especially if i can't use the modern features i'm used to. I don't even know if Mocca is actually able to process esm. It wasn't last time and together with polluting the global namespace, i decided that it was the first and last time i deal with moccas nonsense.

I don't understand how anyone can work on a project when the following is the output of a simple install. I at least can not, and it is so excrutiating to fix things like these.

My apologies for this short rant. Again i want to express that I'll gladly answer any questions about my fix or about modernizing parts of the project. And if maintainers are in favour of modenizing, I will gladly join in the work, i just won't do it alone only to find out the maintainers are not interested in upgrading because "it works just fine as it is".

~/d/G/chevrotainnpm clean-install                                                                           
npm WARN deprecated mkdirp-promise@5.0.1: This package is broken and no longer maintained. 'mkdirp' itself supports promises now, please switch to that.
npm WARN deprecated request-promise-native@1.0.9: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated har-validator@5.1.3: this library is no longer supported
npm WARN deprecated flat@4.1.0: Fixed a prototype pollution security issue in 4.1.0, please upgrade to ^4.1.1 or ^5.0.1.
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated mkdirp@0.3.0: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated coffee-script@1.12.7: CoffeeScript on NPM has moved to "coffeescript" (no hyphen)
npm WARN deprecated core-js@2.6.11: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.

added 2102 packages, and audited 2109 packages in 2m

72 packages are looking for funding
  run `npm fund` for details

17 vulnerabilities (15 low, 2 high)

To address issues that do not require attention, run:
  npm audit fix

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.

NOTE: Unsupported engine is from fs-extra which will not even run in node-v14

npm WARN Error: Unsupported engine 
npm WARN deprecated request-promise-native@1.0.9: request-promise-native has been deprecated because it extends the now deprecated request package, see https://github.com/request/request/issues/3142
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
npm WARN deprecated chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm WARN deprecated mkdirp@0.3.0: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142

added 1457 packages, and audited 1457 packages in 18s

53 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

Hello @KilianKilmister

Thanks for reporting these Issues 👍

Friendly Advice

In general I'd suggest less ranting, assumptions and judgment and more professionalism and understanding when communicating in OSS context.

As a project that has been around for 5-6 years now, you should expect some level of legacy and modernizing needed, It does help when you point out these issues and link to relevant docs, and it would help even more if you can even assist in solving some of these issues. However random complaints/rants are non-productive any may even be counter-productive e.g by de-motivating OSS maintainers from investing their free time in supporting OSS software...

Back to the Technical Topics

Named Imports

I've been able to reproduce the issue by modifying the ECMA6 (commonJS) example. To use .mjs suffix and import syntax.

I will investigate this farther in the future, I am a little wary of the "exports" property being a breaking change, and also I have not seen it actually used yet in a couple of popular libraries (lodash / react), so I wonder how other libraries resolve this issue?

I am also a little wary of your descriptions of the many changes needing to be made, I guess I will be wiser when I have time to dive deeper into it.

deprecation warnings on npm install

The project uses yarn not npm for development purposes.
- https://github.com/SAP/chevrotain/blob/master/CONTRIBUTING.md#initial-setup
- yarn does not show any deprecation warnings on initial deps install.
Most of these deprecations have already been resolved as I've recently merged automated PRs by dependentBot.
The remaining deprecations are transitive dev-deps, I am not sure what I could do about those. It seems to be the responsibility of upstream projects, nor am I sure if this is actually a serious issue as we are dealing with dev-deps and running CI on all modern versions on node-js.

fs-extra and node 14

I am not sure there is actually problem. But this made me inspect the circle-ci config and made me realize The project is running the CI voter on nodejs 10/12/13 instead of 10/12/14. 👍 This was fixed in this commit: https://github.com/SAP/chevrotain/commit/d76d72f1e24f2d61ec4ce2aaf7623af793133c2e which passed successfully on circle-ci:

I also recently updated (via dependentBot) fs-extra from 9.0.0 to 9.0.1. But that does not seem to have change fs-extra node engine version constraint:

regexp-to-ast library

My intent is to deprecate the uthe regexp-to-ast library as I lack the time to maintain it. See: https://github.com/SAP/chevrotain/issues/777

This is actually the highest priority next item I would like to tackle as with the ECMAScript regexp syntax being a moving target, more and more issues and gaps are being detected with the regexp-to-ast libray And this will reduce ~8-9% of the amount of (productive/runtime) source code that needs to be maintained in the scope of Chevrotain.

I am actually more interested in removing / extracting some less commonly used features to reduce the maintenance overhead of Chevrotain, as it is not a small library with ~9,000+ LOC (productive/runtime).

Mocha a tests

I am not familiar with mocha and ESM modules either. The common alternative to mocha is Jest but that was not relevant for this project as:

There used to be browser testing using karma in Chevrotain.
Jest was none existent or in its infancy when Chevrotain was released.
Multiple processes may actually be counter productive in projects using Chevrotain as the high (one time) initialization cost of Chevrotain parsers may be need to paid multiple times (per each process).

IMHO the main issue with the testing is due to the historic mistakes of not encapsulating all test runtime logic (e.g before/after sections). So the end result is that even if only a single test is executed debugging can be difficult as a-lot of none related productive code may be executed as it is called from "describe" blocks. It is obviously very time consuming to fix this now (in hind sight) because there are over 14,000 LOC of tests...

Cheers. Shahar.

Apreciate the lenghty (and fast) response

As there is a couple of things i would like to explain suficiently, I'm going to split it up, writing the more in depth parts in a separate doccument (probably on my fork, i'll comment a link in this issue-thread once its done.). And no worries, these are all just suggestions and opinions. Course of action is up to you.

About the Advice

I'm not proud to say this, but this was the nicest formulation i could muster at that point (explanation following), i also want to stress that nothing i've written above was meant in any demeaning way (exept for the spite thrown at Mocha), and my apologies again. I'm very happy about your potential interest. Modernism vs Conservatism is also a core issue to me, and i've grown weary by repeatedly having invested considerable amounts of work but everything being disregarded without proper reasoning . My work pretty much exclusively is in dev-ops in an esNext environment, it's vital for me to explore modern tools and practices and to be able to abandon something as soon as a more convenient way apears. ESM vs CommonJS for example: i've been using ESM for so long that i have a really hard time working with CommonJS because of how much easier ESM is to use with not having to worry about circular references and being able to use top-level-await etc.

I put a lot of effort in the issues i file, but i can't be spending too much time on a single issue, because i need to get back to my project at some point or discover other unrelated issues while exploring one. So i'm usually quite exhausted at the end and just want to get the report done.

I pulled a few too many all-nighters in the past few weeks because of two projects that are very important to me. On one hand I'm involved in the npm-v7beta where im pushing heavily for a feature, for which i need to work on specs and prototyping dicuss it in meetings, the otherone is a new command-shell with JS/ES like syntax (which causes my interest in this project). Long story short: I need to take some more breaks.

Enough rambeling, let's get down to buisness:

Named imports

I will investigate this farther in the future, I am a little wary of the "exports" property being a breaking change, and also I have not seen it actually used yet in a couple of popular libraries (lodash / react), so I wonder how other libraries resolve this issue?

Well, it is a semver-major. but it also gives an author precise control over what he wants to expose. Using not explicitly mentioned entry points into a module also should pretty much forfeit any expectation of stability, as this is pretty much the definition of internal code.

An example of a popular tool who was a very early adopter is rollup (over 2.3 weekly downloads). They included it in the release of version 2.0.0, and the had it working without any issues one day later and ever since. (original commit, fix 1, fix 2). I will reference them again in the seperate document when making the case for solid ESM support.

From their current package.json

{
  "exports": {
    ".": {
      "node": {
        "require": "./dist/rollup.js",
        "import": "./dist/es/rollup.js"
      },
      "default": "./dist/es/rollup.browser.js"
    },
    "./dist/": "./dist/" // <- folder export
  }
}

I am also a little wary of your descriptions of the many changes needing to be made, I guess I will be wiser when I have time to dive deeper into it.

If we leave away the replacement of fs-extra (which was made on a convenience base only) and "allowSyntheticDefaultImports": true (as the regexp-to-ast might be kicked out, and that's the only thing using it), It isn't really much that would need to change:

file extentions in all relative import statements (that's a single regex replace across the directory and does not change anything already in place)
conditional import to package.json (some care needs to be taken, but you can always aim a bit to broadly to be save)
package.json { "type": "module" } inside the esm-lib (cirrently a simple copy command, doesn't affect anything else either)

All other things are not strictly nescessary.

deprecation warnings on npm install

I red the CONTRIBUTING.md, but i'm not familiar with it so i switched the nescessary configs to use npm.

A lot could be discussed about deprecation and vulnerabilities, but I'll leave that for another time. I want to only adress a single point here: Lerna. Quoting a comment from its issue page: #2703

(comment)

I also wanted to mention #1172 to show that support has been difficult to find for close to 2 years on this package, yet, I haven't seen much of an effort to seek out additional maintainers.

Currently, all maintenance must go through @evocateur , since he is most familiar with the project and is one of two people with access to publish changes. Would love to hear what would be acceptable from the owners of this project to help find a LTS model for this package, instead of relying on 1 person to contribute to it in their free time.

The thread covvers the detials, but right now lerna is in a limbo. Depending on what happens in zje next few weeks, it might be smart to drop it from the project. But i can't really suggest one way or the other as things stand.

fs-extra and node 14

fs extra doesn't really provide anything that node-v14 (or even 12 actually) doesn't have as a builtin. fs-extra is a bit more fool-proof, but looking at your detailed reply I don't think that you need much fool-proofing. I think it would be easiest to just drop it (it wouldn't be much of a problem to rewrite the scrips i made to be usable without top-level-await).

I'l be talking more about node-v14 and related things in the other document.

regexp-to-ast library

Might not be a bad idea to replace it. I ran in to issues because i needed to use unicode property escapes as they are pretty great (/\p{ID_continue}/ expresses far over 2000 chars with minimal overhead and is automatically extended when a new version of unicode arrives, can't do that with character classes). I will be talking about a few related things in the other document.

removing / extracting some less commonly used features

By the layout of the Workspace i'm assuming it's intended to be a mono repo anyways, right? I think it would be a great idea to split off parts by functionality into a few seperate packages, it really is a pretty large codebase for only one. But i'll also be talking more about that in the second document. For some parts it would probably be smart to freeze them until there is capacity to work on them.

Tests

The one i commonly see used is Tap, it's usable for both cjs and esm and is imported/required like any other module. It also doesn't cloud up controle over the test scripts. each file could in itself be a test that could be run alone with node path/to/file.js and it can be used with ts-node, so it doesn't need prior compilation of the entire project all the time.

Depending on the tests to be run, it might also make sense to run tests against a server, passing the necessary information to it and validating the reply. Such a server is easy to implement on a socket/pipe/port and if applicable, tihs would solve much of the overhead issue. Similarly, using a docker container might be smart, but both of these are not very important right now. I can talk more about it if you want.

What exact method makes sense will also depend on the potential splitting/shaving parts of the project

This has gotten way longer than i wanted it to be, but so be it. It will probably be a few days before i got the other one finished. I'm planning to take a well needed break.

@KilianKilmister no worries, and no harm done 😄 and reply whenever you have time (after some R&R).

Named Imports.

I am interested in solving this, however its not something I can tackle immediately. The documentation mentions issues with package level state:

https://nodejs.org/api/esm.html#esm_approach_2_isolate_state

I think this may make the impact of an existing bug #1056 worse. So this global package level state may need to refactored away first.

It's fine to make breaking changes, as long as they are well documented.

https://sap.github.io/chevrotain/docs/changes/BREAKING_CHANGES.html

However sometimes its best to group together multiple breaking changes so that may also affect timelines.

Lerna

Interesting that it is in limbo, But is there any alternative?

fs-extra and node 14

I had a quick attempt at removing fs-extra, and in fact it is only really needed for a copySync operation.

of directories.

While its a little ugly to have the whole fs-extra dependency for such small functionality It may be preferable from writing our own recursive folder copying.

Unless I am missing some built-in capability in nodejs fs package?

regexp-to-ast library

This library is part of an optional optimization flow. So you won't be losing any functionality in Chevrotain If it can't parse your regexps, only performance, which will be re-gained once I replace regexp-to-ast with a better regexp parser (hopefully).

https://sap.github.io/chevrotain/docs/guide/performance.html#ensuring-lexer-optimizations

removing / extracting some less commonly used features

Initially this was a single repo with a single productive package and examples which used home-brewed scripts that did npm link.

At some point I've refactored it to a lerna mono-repo with a single published package and the examples as private packages.

The next step would be to start splitting apart the functionality into multiple smaller packages. However this would of course need to be done incrementally due to capacity constraints.

Tests

It seems like the esm source code is tested with mocha as well using the esm package wrapper.

"test:esm": "mocha \"./lib_esm/test/**/*spec.js\" --require esm"

I have been using Mocha in multiple projects for many years and I'm pretty fond of it. I actually do like the separation in that it does not include an assertion library or even coverage functionality, it does allow for more versatility but at the cost of configuration headaches 😢 .

The concept in Tap of each test file being directly executable is very appealing. In pure JavaScript project I enjoy a similar functionality (direct execution) via the IDE as JetBrain's WebStorm adds little "play icons" next to Mocha "it"/"Describe" calls. Perhaps I should try tap in my next project,

I wonder how Tap deals with TypeScript projects. My experience with ts-node is that it did not work well for TypeScript mono-repos with TypeScript project references. I also had other strange issues with ts-node. My current preference for TypeScript Projects is to have a separate compile and test flows:

compile:watch script that re-compiles the whole mono-repo at need.
test script, with hopefully decent IDE integration.

My even stronger preference is to avoid TypeScript code generation and only use TypeScript as an API/Type-Check tool. I'd rather write pure JavaScript with multiple small packages in a mono repo and 100% test coverage as that would gain a faster feedback loop and still enable reasonable refactoring and long term maintainability.

Cheers. Shahar.

Named Imports

I am interested in solving this, however its not something I can tackle immediately. The documentation mentions issues with package level state:

I did check through the code but must have missed that. But either way, It's mostly stateless. but it's great you are pointing that out, as these things could case real issues if missed.

However sometimes its best to group together multiple breaking changes so that may also affect timelines.

Of course. This should be included with changes that would result in a semver-major anyways

Lerna

Interesting that it is in limbo, But is there any alternative?

The short answer is no. The longer answer is: there isn't any tool that fits as broadly as Lerna tries it. the major package-managers, namey pnpm, yarn and npm (only in v7 which is currently in beta), do have some sort of workspace feature, but i'm only familiar with npm and that's only in stage1 so not too feature rich, it is pretty great thogh: the root package defines an array of paths (can use globs). child-packages remain completly independent packages, all their deps will be stored in the root node-modules (works cause of nodejs resolution). Child-packages are symlinked and can thus import eachother by name, which means they can also be published and used independently (their siblings now being normal dependencies that get auto-installed). this makes things more flexible than the yarn implementation. pnpm is marketed as monorepo tool, but i never used it and don't know much about it. The most common aproach is still just writing the neccessary scripts tailored to the usecase, but this obvously workintensive and requires expertise.

fs-extra and node 14

I've been meaning to ask: any specific reason you used synchronous filesystem operations? Or are you just not used to their async counter-parts and wanted to be save?

Also, as this made me realize i didn't have a function to copy a directory in my personal utility lib, i wrote some up a sync and async one and i wanted to do some benchmarks just for fun (and to learn the node API). sync vs async doesn't make much of a difference for the occasional use, but async tends to be somewhere close two twice the speed.

but this isn't an urgent thig, so it's not too important.

tests

the esm package has a slightly different interop between CommonJS and ESM (gh issue). this can be quite dangerous as all the tests could pass while the actual run will fail on startup.

ironically, if you want to use it with tap you have to set its options to { esm: false } or else it would also use the esm package. but once you do that, it works.

I never used webstorm, I'm using vs-code. it also has things like these, but i don't know how much they differ.

ts-node can be a bit of a pain to get working correctly, but if you know the pitfalls, it works perfectly. tap can precompile and test typescript, i believe, but as it's just another module you can just start the file with ts-node and and have it load any typescript you want.

ts-node also also for time has been working on a native esm loader. It's still experimental, but i've been using it since the start and it works like a charm. Really removes the layer of abstraction that could be felt before. (ts-node/esm thread)

with the native loader all you need to do is start node with:

node --loader ts-node/esm path/to/file                             # with typechecking
node --loader ts-node/esm/transpile-only path/to/file    #just run

on some unix distros you can add command arguments to the hashbang (macOS works, linux doesn't for exampel), so i just have the loader arg in the hashbang. eg: #!/usr/bin/env node -loader ts-node/esm

I'm running both proper tests and prototyping stuff with it.

My even stronger preference is to avoid TypeScript code generation and only use TypeScript as an API/Type-Check tool. I'd rather write pure JavaScript with multiple small packages in a mono repo and 100% test coverage as that would gain a faster feedback loop and still enable reasonable refactoring and long term maintainability.

Since i transpile to esNext and esm, the resulting JS/ES code is pretty much exactly like the source with all the type information stripped away. And this works for small things just aswell to be honest. i mean i write my build scripts in typscript, too.

i'm currently trying out const enums, since they are transpiled away, but are really great for documentation and make consistent changes easier across the procect.

[...] as that would gain a faster feedback loop and still enable reasonable refactoring and long term maintainability.

With a project architecture like i'm using, the difference is about 1-3 extra seconds spent on transpiling before any run (being done automatically). this becomes negligable in my opinion considering all the added safety. Both in classic repos and in mono-repos. One of the projects i'm working on is a mono-repo with the smallest package containing just under 100 LOC, a tsconfig.json (that extends and is referenced by the root tsconfig.json) and a package.json. as it's a private package used for building, it's not test covered, but tap is perfectly able to handle monorepos.

For example the following TS source file file from a small package in a mono-repo of mine containing a few files. (it's by far the largest file in that package)

import { Null } from './null.js'

export interface TriggerGroup extends Trigger {
  children: Trigger[]
}

export class Trigger<F extends (...args) => any = (...args) => any> extends Null {
  #isArmed = true
  get isArmed () { return !!this.#isArmed }
  trigger: (...args: Parameters<F>) => boolean
  disarm: () => boolean
  constructor (action: F) {
    super()
    if (action instanceof Trigger) return action
    this.trigger = (...args: Parameters<F>) => {
      if (this.#isArmed) action(...args); return this.armed
    }
    this.disarm = () => { const state = this.armed; this.#armed = false; return state }
  }

  static makeGroup (...actions: Array<() => any>) {
    return this.link(...actions.map(action => new Trigger(action)))
  }

  static link (...triggers: Trigger[]) {
    const triggerGroup = triggers.filter(trigger => trigger instanceof Trigger)

    const trigger = new Trigger((...args) => triggerGroup.some(child => child.trigger())) as TriggerGroup
    const { disarm } = trigger
    trigger.disarm = () => disarm() && triggerGroup.some(child => child.disarm())
    trigger.children = triggerGroup
    return trigger
  }
}

Turns into this JS/ES file. this makes it explorable in the transpiled version aswell, as all the doc comments would be included. just one of the reasons why esm is amazing.

import { Null } from './null.js';
export class Trigger extends Null {
    #isArmed = true;
    get isArmed() { return !!this.#isArmed; }
    trigger;
    disarm;
    constructor(action) {
        super();
        if (action instanceof Trigger)
            return action;
        this.trigger = (...args) => {
            if (this.#isArmed)
                action(...args);
            return this.armed;
        };
        this.disarm = () => { const state = this.armed; this.#armed = false; return state; };
    }
    static makeGroup(...actions) {
        return this.link(...actions.map(action => new Trigger(action)));
    }
    static link(...triggers) {
        const triggerGroup = triggers.filter(trigger => trigger instanceof Trigger);
        const trigger = new Trigger((...args) => triggerGroup.some(child => child.trigger()));
        const { disarm } = trigger;
        trigger.disarm = () => disarm() && triggerGroup.some(child => child.disarm());
        trigger.children = triggerGroup;
        return trigger;
    }
}
//# sourceMappingURL=trigger.js.map

While I have no association with Lerna, I would indicate that one way that could help keep that project alive would be to have additional contributors help out there. I'm not sure how the primary maintainer would like to manage that, but, I suspect getting assistance on PRs and issue fixes might help based on the limited emails I've had with them. That being said, PRs and issues seem to go for months without any feedback, so, I suspect it will require someone to take it over completely, at some point.

@MikeActually I've read through the relevant discussions about this a few days ago, and we will have to see what the result of the dispute claim will be.

That being said, PRs and issues seem to go for months without any feedback, so, I suspect it will require someone to take it over completely, at some point.

I can say right of the bat, until something changes that, i'm not going to do any work on lerna.

I'm going to continue this in your issue thread in the lerna repo because it's not the topic of this thread.

Only wanted to let you guys know that node now got a experimental specifiers flag so you do not need the file extension https://nodejs.org/api/esm.html#esm_import_specifiers

@frank-dspeed It had it for quite a while, but i think they aren't a good practice to use. if specifiers are included in an import statement, anyone will be able to tell what kind of file is requested. I'd advise against people using the old practice, as adding a file extention is a negligable amount of work.

Back to working a bit on OSS this weekend 😄

@KilianKilmister

Lerna

The most common aproach is still just writing the neccessary scripts tailored to the usecase, but this obvously workintensive and requires expertise.

I used to write my own scripts for a pseudo mono-repo. replacing them with Lerna + yarn worked much better 😄. I will keep an eye out on Lerna state, but there would have to be actual problems in this repo's work flows and / or a "new and better" alternative for any switch. I'd rather not go back to home brewed scripts for this purpose.

Scripts

I've been meaning to ask: any specific reason you used synchronous filesystem operations?

In the context of small dev scripts that work on small data sets it is the most simple approach.

It does not matter if the script takes 10-20-40 mili second no need to optimize for performance
No need to Avoid "locking" the process (no separate GUI logic that becomes unresponsive).
No need to deal with wrappers for lack of top level await (scripts must run in node 10 too).
...

Project Structure and ts-node

@KilianKilmister: I did not understand from your reply how you got ts-node to work with TypeScript project references and/or incremental builds.

With project references and incremental build I am getting ~1 second compilation times for small changes which is pretty good:

Using a 2018 I7 laptop
On a not small 20K Loc project:
https://github.com/SAP/ui5-language-assistant/blob/master/tsconfig.json

Regarding the reduction of the abstractions. Compiling to a higher target of ECMAScript is indeed something I am looking forward to perhaps even in version 8.0.0 where I may finally deprecate IE11 support 😄

However all abstractions may be leaky at some point. And I tend to regard ts-node as another leaky abstraction (see issues linked above and also your comment about it being a bit painful...). And prefer the simplicity and separation of concerns in the tooling (compile vs testing vs coverage vs ...).

I even regard TypeScript as a sometimes leaky abstraction, which is why in a small project I may consider writing the code in pure ECMAScript and only the APIs/signatures in TypeScript.

This is of course a subjective opinion. 😄

@bd82 I believe you have to add "tsnode": { "files": true }to the tsconfig so it loads the necessary definitions. i think it might be nescessary to register the loader on node directly (so not using the standalone ts-node command), but i'm not 100% sure, all i know is that it works in the configuration i'm using, lol. I like it because it makes it easy to run files that would cause issues in a proper build.

But this isn't really that important, as long as a working way exists.

Compiling to a higher target of ECMAScript is indeed something I am looking forward to perhaps even in version 8.0.0 where I may finally deprecate IE11 support 😄

Hearing things like this make me really happy, can't wait for the day were IE11 will be nothing but a memory. Node-v10 as the current Maintenance LTS (EOL in 2021-04-30) would support up to and including ES2017 and unless you want to use a few of the higher proxy traps, even ES2018

However all abstractions may be leaky at some point. And I tend to regard ts-node as another leaky abstraction (see issues linked above and also your comment about it being a bit painful...).

That's true. "the lesser of two evils" is also subjective of course.

I will keep an eye out on Lerna state, [...]

Sure, i think that's the right thing to do for the time being.

In the context of small dev scripts that work on small data sets it is the most simple approach.

I wanted to benchmark my own functions against fs-extra and funily it turnes out the async copy function of fs-extra is by far the slowest of what i tested, about three times slower than both mine and their sync version and about 6 times slower than my async, lol.

Is there anything we talked about for whitch you would like to see a mock-implementation or PoC? We talked about pretty much everything i wanted to discuss except for some very specific technical aspects, which don't really need discussing unless they become relevant. Otherwhise i would propably just start hacking away on my fork and see what neat stuff i can do.

Chevrotain / chevrotain