Bundle CLI using webpack

[molant]

As mentioned in #896, one of the things we can do is to bundle everything with webpack. This will reduce the installation time and should improve the boot time.

Here's the branch with the work

Some of the things we need to take into account and figure out:

• [x] With the work to lazy load depending on the task (#1101) do we we need to create different chunks or do something especial to take advantage of it or webpack will do some magic?
• [x] We do some dynamic imports with require in the code (when loading rules/parsers/formatters, etc., load package.jsons from packages, etc.). The guidance I got from @TheLarkInn is to wrap those in new Function or eval so webpack doesn't try to optimize those
• [ ] We probably want to optimize some of our import statements so we can get better 🌳 shaking (just import what we need instead of *)
• [ ] Add shebang to the output and/or a loader
• [x] Figure out how to change the strategy to load .hbs depending if we are bundling or not. Solution: Set a webpack environment variable that does a require or the other method.
• [ ] Something else?

The initial tests show an output of 2.16MiB which is not bad at all (and we can probably optimize further):

[molant]

When we execute the bundle, the output is:

node dist/cli.js -v

webpack:///external_%22update-notifier%22?:1
module.exports = update-notifier;
                 ^

ReferenceError: update is not defined
    at eval (webpack:///external_%22update-notifier%22?:1:18)
    at Object.update-notifier (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:23511:1)
    at __webpack_require__ (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:20:30)
    at Object.eval (webpack:///./src/lib/cli.ts?:7:73)
    at eval (webpack:///./src/lib/cli.ts?:92:30)
    at Object../src/lib/cli.ts (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:22840:1)
    at __webpack_require__ (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:20:30)
    at eval (webpack:///./src/bin/sonarwhal.ts?:4:66)
    at Object../src/bin/sonarwhal.ts (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:22828:1)
    at __webpack_require__ (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:20:30)

The weird part to me is that it complains about update and not update-notifier. If I change module.exports = update-notifier; to module.exports = 'update-notifier'; in the code then it will complain about require-uncached. Yet another package with - in the name, but this time it is uncached that can't be found. Doing the same "hack" will output the following:

webpack:///./src/lib/utils/misc.ts?:260
        throw new Error('No package found');
        ^

Error: No package found
    at findPackageRoot (webpack:///./src/lib/utils/misc.ts?:260:15)
    at findPackageRoot (webpack:///./src/lib/utils/misc.ts?:262:12)
    at Object.eval (webpack:///./src/lib/utils/handlebars.ts?:14:113)
    at eval (webpack:///./src/lib/utils/handlebars.ts?:49:30)
    at Object../src/lib/utils/handlebars.ts (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:23164:1)
    at __webpack_require__ (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:20:30)
    at Object.eval (webpack:///./src/lib/cli/wizards/new-rule.ts?:21:76)
    at eval (webpack:///./src/lib/cli/wizards/new-rule.ts?:359:30)
    at Object../src/lib/cli/wizards/new-rule.ts (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:22948:1)
    at __webpack_require__ (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:20:30)
    at eval (webpack:///./src/lib/cli/actions.ts?:3:75)
    at Object../src/lib/cli/actions.ts (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:22852:1)
    at __webpack_require__ (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:20:30)
    at Object.eval (webpack:///./src/lib/cli.ts?:12:70)
    at eval (webpack:///./src/lib/cli.ts?:92:30)
    at Object../src/lib/cli.ts (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:22840:1)
    at __webpack_require__ (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:20:30)
    at eval (webpack:///./src/bin/sonarwhal.ts?:4:66)
    at Object../src/bin/sonarwhal.ts (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:22828:1)
    at __webpack_require__ (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:20:30)
    at C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:69:18
    at Object.<anonymous> (C:\projects\@sonarwhal\sonarwhal\packages\sonarwhal\dist\cli.js:72:10)

which at least is an error sonarwhal throws.

@TheLarkInn the issues with names that have - could be a problem with webpack?

[molant]

I've pushed a new commit that fixes the - modifying the exports:

externals: {
  browserslist: 'browserslist',
  encoding: 'encoding',
  'require-uncached': 'commonjs require-uncached',
  'update-notifier': 'commonjs update-notifier'
},

But now we need to figure out how to load the .hbs for the wizards. I'm currently using handlebars-template-loader but I'm not sure how to do a fallback for when we are not using webpack (I'll add this to the list in the top).

Now when I execute it I get the following:

webpack:///external_%22browserslist%22?:1
module.exports = browserslist;
                 ^

ReferenceError: browserslist is not defined

It could be because we have a browserslist and also a package with the same name.

[molant]

I've worked a bit this morning with @TheLarkInn and there's now a new version that generates chunks and has a few more things working (basically all the options minus actually analyzing a site).

Things to figure out:

• [ ] With the new work, there are 20 bundles created automatically and 14.4 MB. That's quite a lot compared to the initial and I'm not sure why. @TheLarkInn any ideas?
• [ ] Can't load the connectors. Probably because they are not a separate package, we generate the name dynamically and webpack doesn't include them. Might be time to separate the connectors from the CLI as well. (FYI @alrra)
• [ ] Need to figure out how to export the utils that are used by the resources. Maybe a new bundle defined manually? @TheLarkInn throughts?
• [ ] Testing, testing, testing

[molant]

I'm working on separating part of the functions we have under utils in the main package into its own subpackage or otherwise it will not be possible to bundle everything correctly (some circular references when bundling...). These are the ones I've identified. Some will go to a new package, others to existing ones. @alrra, @sarvaje if you have any preference please let me know.

Function	New package/folder	Used By
async-wrapper	utils-common/misc	http-compression, no-bom
caniuse	utils-common/misc	disown-opener, manifest-is-valid, meta-theme-color
content-type	utils-common/network
debug	utils-common/misc	Everyone
handlebars-utils	utils-common/misc	create packages
logging	utils-common/misc	Everyone
hint-helpers	utils-tests-helpers?
resource-loader	stay in hint somewhere else?
schema-validator
fs/file-extension	Maybe delete it?	manifest-file-extension only
fs/is-file		connector-local, and then network/as-uri
fs/load-json-file		no-vulnerable-javascript-libraries, webpack-config, serveral parsers
fs/read-file	Try to switch to -async?	create-hint, amp-validator, performance-budget, sri, parser-typescript, parser-babel
fs/read-file-async		connector-chrome, connector-local, create-parser, axe, no-vulnerable-javascript-libraries
fs/write-file-async		connector-chrome, create-hint, create-parser, no-vulnerable-javascript-libraries
misc/cut-string
misc/delay
misc/generate-html-page	utils-tests-helpers
misc/normalize-string
misc/normalize-string-by-delimiter
misc/to-camel-case
misc/to-lowercase-keys
misc/to-pascal-case
network/as-uri
network/as-path-string
network/is-html-document
network/has-attribute-with-value	maybe delete?	utils-debugging-protocol-common
network/is-http
network/is-https
network/is-local-file
network/normalized-header-value
packages/is-official
network/is-regular-protocol
packages/load-package
network/request-async

[molant]

Finally we are going to bundle all the create- packages but not the main CLI. A few reasons:

• there are many utils in the main CLI that are used by other packages which made the bundling and configuration weird
• separating the utils into its own packages wasn't very straightforward with some nasty dependencies that caused circular references and not a lot of ROI after the improvements in #896