Simplify code generation, build scripts, and contributor scripts

[strager]

quick-lint-js has some complicated unorthodox tools which are required for some unorthodox artifacts or which are just used by contributors:

• used during C++ build:
  • **/CMakeLists.txt (CMake; calls C++ toolchain)
  • tools/collect-copyright (optional) (Python; calls system package managers; relies on C++ linker output)
  • test-cli.py (optional) (Python; calls quick-lint-js CLI)
• used to make unorthodox artifacts:
  • tools/browser-globals/ (Node and yarn; needs web-specs Git repo)
  • tools/generate-lex-keyword (Bash; uses gperf)
  • tools/generate-lex-unicode/index.js (Node and yarn)
  • tools/update-translator-sources and tools/compile-translations.go (Bash and Go; calls gettext commands)
• used for maintaining the website and docs:
  • docs/man/generate-man-pages (Python and Ruby; installs and calls asciidoctor)
  • tools/add-version-numbers-to-error-docs.py (Python)
  • tools/check-links (Python; needs requests and bs4 Python packages)
  • tools/generate-vim-tags (Bash; calls Vim)
  • website/ (Node and yarn; needs C++ WASM build)
• used to keep the code clean:
  • plugin/vim/test-nvim.sh (Bash)
  • plugin/vim/test-vim.sh (Bash)
  • tools/check-file-copyrights (Bash)
  • tools/fix-include-guards.go (Go)
  • tools/format (Bash; calls clang-format; calls fix-include-guards.go)
• used for shipping a release:
  • dist/arch/lint.sh (Bash)
  • dist/arch/update-aur.sh (Bash)
  • dist/chocolatey/build.go (Go)
  • dist/debian/package.sh, dist/debian/build.sh, dist/debian/strict-lintian.sh (Bash)
  • dist/debian/sync-releases-to-apt (Bash)
  • dist/debian/update-repository (Bash)
  • dist/release.go (Go)
  • dist/scoop/make-manifest.go (Go)
  • dist/sign-release.go (Go; calls Relic and shasum tools)
  • dist/winget/make-manifests.go (Go)
  • plugin/vim/create-archive (Bash)
  • website/deploy.sh (Bash)
• extra stuff used only manually:
  • benchmark/benchmark-lsp/ (C++; uses quick-lint-js APIs)
  • infrastructure/quick-lint-js-web-2/ (Ansible)
  • src/quick-lint-js/debug/ (Node and yarn)
  • tools/analyze-trace.cpp (C++; uses quick-lint-js APIs)
  • tools/build-sizes/aggregate-binary-sizes (Python)
  • tools/build-sizes/collect-many-binary-sizes.go (Go)
  • tools/build-sizes/collect_binary_sizes.py (Python; calls LLVM tools and Debian tools)
  • tools/build-sizes/compare-builds.py (Python)
  • tools/build-sizes/index.html (HTML + JavaScript)
  • tools/check-esprima (Python; bit rotted)
  • tools/check-test262.go (Go)
  • tools/ci-analytics/load_github_actions.go (Go)
  • tools/create-translation (Bash; calls gettext commands)
  • tools/lines-of-code (Bash)
  • tools/test-all-js-engines (Bash; calls npm to install jsvu)
  • tools/test-line-count.py (Python)
  • tools/test-translation (Bash)

My issues:

• Too many languages: Bash + C++ + Go + Node.js + Python
• Several third party dependencies, some of which are undocumented
• Most tools themselves are undocumented
• No centralized way to update unorthodox artifacts (e.g. Makefile)
• Hard to share code between scripts
• Some scripts are Linux-specific or POSIX-specific

Related: #168

[strager]

Languages

For core stuff, I think C++ and JavaScript are here to stay. C++ might be replaced with Rust if Rust build times improve. JavaScript is the language of web tooling, and I think that's fine.

For scripting, we currently use Ansible, Bash, Go, Node.js, and Python:

• Ansible: Verbose and slow. Arguably better than shell scripts though due to forced organization.
• Bash: Nice for quick automation, but gets ugly quickly. Bash scripts don't work well on Windows. But Bash is decent if we know we're on Linux (e.g. Arch Linux and Debian scripts).
• Go: I introduced Go due to performance problems with Node.js and Python, and static typing is nice for scripts. Go doesn't support shebangs, though. I haven't tried using third-party dependencies with Go, so I don't know if it's meh like Python or tolerable like Node.js.
• Node.js: We JavaScript for the website, but do we need it for anything else? It might be a good idea to limit JS to just website-related things.
• Python: Many people already have Python installed, making it a nice choice for the C++ build (tools/collect-copyright). But perhaps we could make the lives of self-builders and CI a bit harder by switching to Go or Node.js. (Bash is not an option because of Windows.)

Other scripting options:

• C++: Monolanguage is appealing. But C++ is a headache for things like execing processes or juggling strings. This can be mitigated by making or finding a library though.
• Haskell: We used to have some Haskell code (for benchmark/benchmark-lsp/), but I replaced that with C++. I think I won't introduce Haskell again.
• Java/Kotlin: I suspect JVM languages would be bad for scripting, but what do I know?
• Perl: I don't know Perl, and I heard it's a pain on Windows. Perl has been losing popularity.
• Ruby: I don't know Ruby, and installing dependencies is meh.
• Rust: I don't know how well-suited Rust is for scripting. Perhaps there's some shebang support?

I'm leaning toward this resolution:

• Create a proper utility package for Go scripts.
• Drop most Bash; replace with Ansible and Go.
• Drop Python; replace with JavaScript (tools/check-links) or C++/Go (tools/collect-copyright) or Go (everything else).
• Keep JavaScript for tools/browser-globals/ and tools/generate-lex-unicode/ only. (I'm very uncertain about this, though. Maybe I could port to Go?)

[strager]

Documentation

• If I have a use case, how can I find the appropriate tool?
• If I see a tool, how can I learn what it does?
• If I see a tool, what other programs do I need installed to run it?
• If I see an unorthodox artifact, how do I regenerate it?
• If I see a source file, how do I know whether it's unorthodox or not? Do I need to care?

Some ideas:

• Document workflows. Workflows will mention tools.
  • Example: 'How to add a new diagnostic.' This document would mention tools/update-translator-sources and the GNU gettext dependency.
  • Example: 'How to submit a patch.' This document would mention tools/format and the clang-format version issue.
• Make tools self-documenting.
  • Add a comment/string at the top of each tool's main source file. Mention the workflow.
  • Add --help output to each tool.
• Make each tool check its third-party dependencies up-front before doing work (but after --help processing).
• Make unorthodox artifacts self-documenting.
  • Include a comment at the top mentioning the workflow and the specific command to re-generate the file.

[strager]

I investigated Go a bit. I ran into an issue.

By default, if you use a command like go run, then the toolchain will check if a vendor directory exists. If vendor does exist, Go will only load dependencies from that directory. If vendor doesn't exist, Go will download and cache dependencies if needed.

quick-lint-js has a directory called vendor which confuses Go. To work around this confusion, you can pass -mod=readonly when running a go command, making the toolchain ignore the vendor directory.

An alternative to -mod=readonly is go mod vendor, which populates the vendor directory. Unfortunately, this command also deletes everything else in the vendor directory. Goodbye C++ code. You can work around this by undeleting the files, but this clearly isn't what the Go developers intended you to do.

[strager]

Thinking about it more, maybe I should choose Node.js for the unorthodox tools. We would then have only two languages in quick-lint-js: C++ and JavaScript. I'm not sure Node.js will work well though; I'll have to play around with it.

Note: Ubuntu 18.04 (the crustiest distro we support) has Node.js version 8 in its repos.

[strager]

Some things to investigate for Node.js:

• Yarn 2 zero-installs
• Yarn workspaces
• npm workspaces (same as Yarn?)
• Partially committing node_modules
• website/ using code from in tools/ (for example) without package.json fanfare
  • Perhaps this can be solved by adding code directly into node_modules? Would Yarn get angry?

[strager]

I boiled it down to two options for unorthodox tools: C++ or Node.js.

I am experimenting with C++. I replaced gperf with a custom tool (written in C++), and am integrating it into the build. I plan to release this slowly:

• next release: C++ build tool dependency is optional (e.g. when cross-compiling)
• release thereafter: C++ build tool dependency is required (even when cross-compiling)

If nobody complains about the second trial release, then I will move forward with C++ for the following tools:

• tools/update-translator-sources and tools/compile-translations.go (currently not integrated into build, but would be after the C++ port)
• tools/collect-copyright (currently optional)

[strager]

I ported the translation infrastructure to C++ integrated into the build. It's been nice except for one issue: if a change is made which doesn't affect the unorthodox source, the generator will update the source's timestamp anyway, causing most files to rebuild. I should fix this with a write_file_if_different function.