:airplane: nodejs-esm-starter

Starter project for NodeJs esm packages, with rollup, typescript, mocha, chai, eslint, istanbul/nyc, gulp, i18next

:crown: This starter was created from the information gleaned from the excellent suite of articles written by 'Gil Tayar': Using ES Modules (ESM) in Node.js: A Practical Guide (Part 1), which I would highly recommend to anyone wishing to get a full understanding of ESM modules with NodeJS and provides the full picture lacking in other offical documentation sources/blogs. The following description contains links into the relevant parts of Gil Tayar's blog series.

:gift: package.json features

:gem: ESM module

  "type": "module",

:mortar_board: See: Using the .js extension for ESM

This entry makes the package an esm module and means that we don't have to use the .mjs extension to indicate a module is esm; doing so causes problems with some tooling.

:gem: The 'exports' field

  "exports": {
    ".": "./src/main.js"
  },

:mortar_board: See: The 'exports' field

The correct way to define a package's entry point in esm is to specify the exports field and it must start with a '.' as illustrated.

Using the exports field prevents deep linking into the package; we're are restricted to using the entry points defined in exports only.

:sparkles: Self referencing

  "exports": {
    ".": "./src/main.js",
    "./package.json": "./package.json"
  },

:mortar_board: See: Self referencing

This means we can use the name of the package on an import instead of a relative path, so a unit test could import like so:

import starter from 'nodejs-esm-starter'

However, there is still an issue with self referencing like this. typescript will appear not be able to resolve that the package name, but in reality there is no problem. Therefore, we need to disable the resultant error. This is achieved at the import site with a typescript directive as illustarted below:

// @ts-ignore
import starter from 'nodejs-esm-starter'

But that now throws up another issue. What we find now is that when we go to lint the project (just run npm run lint), we'll simply be served up an error message of the form:

4:1 error Do not use "@ts-ignore" because it alters compilation errors @typescript-eslint/ban-ts-comment

It is safe to disable this and we do so by turning off the ban-ts-comment rule in the .eslintrc.json config file inside the "rules" entry:

"@typescript-eslint/ban-ts-comment": "off",

:sparkles: Multiple exports

This starter does not come with multiple exports; it would be up to the client package to define as required, but would look something like:

  "exports": {
    ".": "./src/main.js",
    "./red": "./src/main-red.js",
    "./blue": "./src/main-blue.js",
    "./package.json": "./package.json"
  },

:mortar_board: See: Multiple exports

:sparkles: Dual-mode libraries

This allows the module to be required synchronously by other commonjs packages or imported asynchronously by esm packages. This requires transpilation which we achieve by using rollup.

The '.' entry inside exports is what gives us this dual mode capability:

  "exports": {
    ".": {
      "require": "./lib/main.cjs",
      "import": "./src/main.js"
    },

:mortar_board: See: Dual-mode libraries

NB: we write our rollup config in a .mjs file because rollup assumes .js is commonjs, so we are forced to use .mjs, regardless of the fact that our package has been marked as esm via the package.json type property.

:page_with_curl: The 'files' entry

  "files": [
    "dist"
  ],

This dist entry should be changed to include those items required to be included in the package archive contents (see files for more details).

:mortar_board: See: Transpiling with Rollup

Required for dual-mode package.

:gem: NPM scripts

:open_file_folder: Boilerplate project structure

All rollup related funcitonality is contained within the rollup folder. Currently, there is a separate file for development and production. The main difference between the production and development rollup configs is that for the former, we use the terser plugin to mangle the generated javascript bundle.

:books: options/production/development

The setup is structured to keep the gulp config encapulated away from the rollup config. This means that the user can discard gulp if they so wish to without it affecting the rollup. The flow of data goes from the root, that being rollup/options.mjs, to either rollup.development.mjs or rollup.production.mjs dependending on the current mode which is then finally imported into the gulp file gulpfile.esm.mjs.

It is intended that the user should specify all generic settings in the options.mjs file and export them from there. This way, we can ensure that any properties are defined in a single place only and inherited as required. Clearly, production specific settings should go in the production file and like-wise for development.

:beers: gulp file (gulpfile.esm.mjs)

In order to simplify usage of gulp in the presence of the alternative gulfile name being gulpfile.esm.mjs (as opposed to the default of simply being gulpfile.mjs), a symbolic link has been defined from gulpfile.mjs to gulpfile.esm.mjs. This means that the user can run gulp commands without having to explicitly define the gulp file gulpfile.esm.mjs.

:heavy_plus_sign: Copying resources

The gulp file, contains an array definition resourceSpecs. By default it contains a single entry (copies i18next translation locales) that illustrates how to define resource(s) to be copied into the output folder. Each entry in the array should be an object eg:

  {
    name: "copy locales",
    source: "./locales/**/*.*",
    destination: `./${roptions.directories.out}/locales/`
  }

A copyTask is defined composed from a series of tasks defined by resourceSpecs. If no resources are to be copied, then just remove this default entry and leave the array to be empty.

:rocket: Using this template

After the client project has been created from this template, a number of changes need to be made and are listed as follows:

• Update the name property inside package.json. Initially it will be set to nodejs-esm-starter. The user should perform a global search and replace inside package.js as there are other entries derived from this name. Ideally, there would be a way in json to be able to cross reference fields, but alas, this is not currently possible.
• add a .env file to the root of the project. This will be used to store secrets when the time comes for performing a release. Initially, the user can simply set the contents to:

GH_TOKEN=ADD-KEY-HERE

This can be taken literally, ie if you don't yet have a personal access token, then set it here to a dummy value

• define resources to copy, if any.
• you will notice that there are unit tests defined for checking i18next setup (language-auto-detect.spec.ts)[test/i18next/language-auto-detect.spec.ts] and at the top of the file, the folling import is present:

// @ts-ignore
import starter from "nodejs-esm-starter";

This is a project self reference, so if the project has been renamed (let's say to widget), then this import statement will no longer be valid, so it should be changed to something like:

import widget from "widget"

• ... and then of course, customise the configs as required.

Bundling with rollup

Maintaining external dependiences

As the project grows, it is inevitable that more dependencies will be accumulated. The user should be aware that as the dependency list grows, if no other course of action is taken, rollup will automatically bundle those dependencies, which is typically not what we want. We use rollup to bundle all internal code, not all of the dependencies, which can easily be resolved externally. For this reason, the user should continuously monitor the contents of both the source and test bundles to make sure that it contains only what it should do. This is less important for the test bundle, because that will not ultimately be delivered to the end user, however, it would cloud the process of reviewing the contents of the test bundle.

rollup allows specification of external entities. The rollup options options.mjs in this template contains a default set of externals for both the source and test bundles, defined at externals.source and externals.test respectively. The user needs to update these externals as appropriate. Sometimes, if a an external is bundled, then a circular reference can occur and the user will see a message in the output such as illustrated below:

Synchronizing program
CreatingProgramWith::
  roots: ["/home/plastikfan/dev/github/snivilization/nodejs-esm-starter/test/banner.spec.ts","/home/plastikfan/dev/github/snivilization/nodejs-esm-starter/test/dummy.spec.ts","/home/plastikfan/dev/github/snivilization/nodejs-esm-starter/test/i18next/language-auto-detect.spec.ts"]
  options: {"moduleResolution":2,"module":99,"resolveJsonModule":false,"allowJs":false,"alwaysStrict":true,"sourceMap":true,"noEmitOnError":true,"esModuleInterop":true,"forceConsistentCasingInFileNames":true,"noImplicitAny":true,"strict":true,"strictNullChecks":true,"skipLibCheck":true,"diagnostics":true,"lib":["lib.es2020.d.ts","lib.dom.d.ts"],"target":7,"configFilePath":"/home/plastikfan/dev/github/snivilization/nodejs-esm-starter/tsconfig.test.json","noEmitHelpers":true,"importHelpers":true,"noEmit":false,"emitDeclarationOnly":false,"noResolve":false}
Circular dependency: node_modules/chai/lib/chai.js -> node_modules/chai/lib/chai/utils/index.js -> node_modules/chai/lib/chai/utils/addProperty.js -> node_modules/chai/lib/chai.js
Circular dependency: node_modules/chai/lib/chai.js -> node_modules/chai/lib/chai/utils/index.js -> node_modules/chai/lib/chai/utils/addProperty.js -> /home/plastikfan/dev/github/snivilization/nodejs-esm-starter/node_modules/chai/lib/chai.js?commonjs-proxy -> node_modules/chai/lib/chai.js
Circular dependency: node_modules/chai/lib/chai.js -> node_modules/chai/lib/chai/utils/index.js -> node_modules/chai/lib/chai/utils/addMethod.js -> node_modules/chai/lib/chai.js
Circular dependency: node_modules/chai/lib/chai.js -> node_modules/chai/lib/chai/utils/index.js -> node_modules/chai/lib/chai/utils/overwriteProperty.js -> node_modules/chai/lib/chai.js
Circular dependency: node_modules/chai/lib/chai.js -> node_modules/chai/lib/chai/utils/index.js -> node_modules/chai/lib/chai/utils/overwriteMethod.js -> node_modules/chai/lib/chai.js
Circular dependency: node_modules/chai/lib/chai.js -> node_modules/chai/lib/chai/utils/index.js -> node_modules/chai/lib/chai/utils/addChainableMethod.js -> node_modules/chai/lib/chai.js
Circular dependency: node_modules/chai/lib/chai.js -> node_modules/chai/lib/chai/utils/index.js -> node_modules/chai/lib/chai/utils/overwriteChainableMethod.js -> node_modules/chai/lib/chai.js

... so to resolve this error, the dependency (in the above case chai) should be added to the list of external dependencies (previously mentioned) that needs to be externalised and thus not bundled.

:globe_with_meridians: i18next Translation ready

This template comes complete with the initial boilerplate required for integration with i18next. It has been set up with English GB (en-GB) set as the default alongside English US (en-US). If so required, this setup can easily be changed and more languages added as appropriate. Please also see how to handle fallbacks in i18next.

If translation is not required, then it can be removed (dependencies: i18next and i18next-fs-backend) but it is highly recommended to leave it in. i18next can help in writing cleaner code eg pluralisation of items referenced in user messages, is particularly useful along with an interesting take on interpolation. The biggest issue for users just starting with i18next is getting used to the idea that string literals should now never be used (see exceptions documented for the eslint-plugin-i18next plugin) and this will be made evident by the linting process; in particular, the user is likely to see violations of the i18next/no-literal-string rule.

The lint gulp task will flag up translation violations and another gulp task i18next has been implemented using i18next-parser, which helps with the process of maintaining translations as the code base evolves.

The i18next/no-literal-string should really only be applied to user facing text content. For this reason, the project has been setup to only apply the rule to typescript files inside the "src" directory and not to unit tests, which would have become too onerous for the user to manage.

:robot: Automated releases

Releases have been automated using gulp's Automate Releases recipe. However, this is just an initial setup. The user should become accustomed with the following concepts:

• keeping a clean commit history with conventional commits
• npm version command. But there is a caveat here. Conventional recommends not using npm version, but to use standard version instead (which is part of conventional-changelog).
• automatic version number bumping Conventional Recommended Bump
• using conventional-changelog-cli to generate a changelog from git metadata
• Making a new GitHub release from git metadata with conventional changelog

To run the full release, just run npm run release. Two methods have been defined for completing an automated release, see the following:

:pushpin: Gulp: this recipe generates and publishes releases (including version number bumping, change log generation and tagging) to gihub. In it's current form, it does not publish to the npm registry, so the user will have to add this to the release chain. The gulp release has been defined as a script named "_gulp:rel"

:pushpin: standard version: this is an alternative to what has been defined in the release gulp task and has been defined in package.json denoted by a script entry named "_standard:rel".

By default, release has been set to use "standard", but this can be switched to use the "gulp" version instead.

It should also be noted that there is a third way (not implemented, but mentioned here for reference), which is to use semantic release.

:construction: Required dev depenencies of note

• :hammer: dual mode package rollup (npm)
• :hammer: platform independent copy of non js assets cpr (npm)
• :hammer: merge json objects (used to derive test rollup config from the source config) deepmerge (npm)
• :hammer: type definitions for file system backend (npm) @types/i18next-fs-backend
• :hammer: i18next eslint plugin (npm)
• :hammer: i18next-parser (npm)

:warning: A note about 'vulnerablities' in dev dependencies

An issue was raised to try and resolve the problem of npm audit reporting so called vulnerabilities (mostly relating to gulp dependencies). However, after a lot of head scratching and many failed attempts to resolve, it was discovered that there is a design flaw with npm audit. This is a widely known issue and very well documented at a blog post npm audit: Broken by Design. It is for the reasons documented here, that there is no need to attempt to resolve these issues. A custom audit package.json script entry has been defined that specifies the --production flag, (just run npm run audit).

:checkered_flag: Other external resources

Here's a list of other links that were consulted duration the creation of this starter template

• Typescript, NodeJS and ES6/ESM Modules
• How to Setup a TypeScript project using Rollup.js by Luis Aviles
• Converting a Webpack Build to Rollup
• How to Create a Hybrid NPM Module for ESM and CommonJS
• Gulp for Beginners (although this is a bit old - circa 2015)

:tv: Youtube:

• Full beginner Gulp setup for SCSS, minifying Javascript, and minifying/webp images