Improve dev experience for writing renderer code. Initially thought that this would also be a good opportunity to investigate bundling the main process code, but decided against it because 1) we don't currently do that so it would be increasing the complexity of this work and 2) previous experiments suggest that Rollup may be a better fit for bundling node code.
things within scope include but are not limited to: supporting TypeScript, auto reload on source changes (renderer only), faster build times.
Reduce tooling complexity and dependency. Things like Babel, Webpack, etc tend to add a lot of maintenance overhead and subject us to keeping up with ecosystems to fix things or maintain compatibility. This work is an attempt to understand if having more control in these processes could be more maintainable for us moving forward, without relying as much on the ecosystem.
Notable changes (so far)
using ESBuild instead of webpack for building renderer code
as of now, the only relevant script for bundling is esbuild.mjs. all configuration and any related build steps are performed there.
as of now, the following commands are available:
build: runs the bundler process in its entirety and output to the static directory. accepts a --mode arg of either dev (default) or prod. as of now prod will include minification, but should do other things when we're ready for that.
watch: runs the bundler process in watch mode, which will create a dev server that serves the static/ directory. accepts a --mode arg of either dev (default) or prod. as of now prod will include minification, but should do other things when we're ready for that.
clean: cleans built assets that are outputted in the static/ directory
example commands:
./esbuild.mjs build: compiles the renderer code and outputs it to the static/ directory
./esbuild.mjs watch: compiles the renderer code incrementally and starts the dev server that serves the static/ directory (defaults to http:/localhost:8000).
in development, the electron main window now loads the dev server url instead of the static main.html file. this allows for live-reloading (note, not hot module reloading or react fast refresh). see the preload/main.js preload script to see how that's done (it's just an eventsource that calls location.reload when a change is detected).
note that live-reloading is only enabled for changes that occur in the src/renderer directory. if you make changes to source code in the main process, background processes, or preload scripts, you'll still have to manually stop restart the server to see those changes.
reorganization of some of the source files. for example,
preload scripts now live in the src/preload directory. we can choose to bundle/process these but for now, we don't because they work in a different environment (access to both browser and Node apis because Node integration is turned on).
removed any direct asset imports from source code (e.g. importing an image or font file into js) and instead updated the source code to refer to the static directory. this way we don't deal with configuring loaders for certain assets right now, which was causing some asset duplication.
this may change in the future, but there aren't any notable benefits to being able to import asset paths directly right now.
removed some static assets that didn't seem to be used by the app. there's a small chance that these are used by some dependencies (e.g. i know https://github.com/digidem/mapeo-settings does rely on some assets existing in the project), so may need to evaluate those changes carefully to make sure i didn't break something subtle.
removal of babel, including the transform that react-intl uses for auto id-ing messages based on file path. as a result, the flow for defining messages is identical to how we do it in Mapeo Mobile. i had to update every call to defineMessages to manually specify something like { id: ..., defaultMessage: ...} for each message string. Based on the lack of a diff in the generated messages/, i think my change has been completed successfully. Just need to keep this in mind moving forward.
changing the main app to load as an ES module instead of requiring the app bundle (see diff for main.html)
Let it be known that I'm not a bundling wizard. It's very possible that the build configuration I've implemented so far needs major reworking based on how browsers and Electron code work. Some other notable issues include:
the pdf report worker doesn't seem to work. if you try to preview the pdf in the observations tab, it just throws and returns the error state. probably something about the way the worker is being bundled that needs to be adjusted, but it's been difficult to debug.
any modules that are specified as external have to be imported via require when used in the main renderer code. for example, most of the node apis are listed as external and can only be used by doing something like const path = require('path'). this is not really a blocking issue, but just a notable change that may be slightly inconvenient.
Remaining work
think it would be useful to consolidate any renderer code that uses node apis in a specific directory in the renderer. for example, if a component does some stuff with the path module such as building a file path via path.join, it should be abstracted to a file located somewhere like src/renderer/node/path.js as a helper function, which is then used by the component. my reasoning for this:
it makes it easier to understand which components rely on node apis. right now, it's a matter of knowing what deps we have that are node-specific and then grepping the codebase to see which renderer components import the various deps. with this re-organization, it will be much easier to just check for files that import from this node-specific directory. this approach probably won't work for all node-specific deps that we use, but if we can cover most of them, i think that's significantly helpful.
having the separation will make it easier to eventually move away from the enabled node integration for the renderer process. it's highly recommended against and there are pretty good references for how to go about it (see https://github.com/electron/electron/issues/9920#issuecomment-575839738), but they require changes in the renderer and main process code.
remove all of the webpack remnants. i haven't done this yet because i wanted a reference to the previous config to make sure the new process covers what we previously had.
tried my best to update the builder.config.js config to make sure that changes in output are accounted for in terms of what needs to be included/excluded when building the electron app. i'm sure i've missed some stuff there since i haven't specifically tested or focused on it yet.
Goals
Notable changes (so far)
esbuild.mjs. all configuration and any related build steps are performed there.build: runs the bundler process in its entirety and output to thestaticdirectory. accepts a--modearg of eitherdev(default) orprod. as of nowprodwill include minification, but should do other things when we're ready for that.watch: runs the bundler process in watch mode, which will create a dev server that serves thestatic/directory. accepts a--modearg of eitherdev(default) orprod. as of nowprodwill include minification, but should do other things when we're ready for that.clean: cleans built assets that are outputted in thestatic/directory./esbuild.mjs build: compiles the renderer code and outputs it to thestatic/directory./esbuild.mjs watch: compiles the renderer code incrementally and starts the dev server that serves thestatic/directory (defaults tohttp:/localhost:8000).main.htmlfile. this allows for live-reloading (note, not hot module reloading or react fast refresh). see thepreload/main.jspreload script to see how that's done (it's just an eventsource that callslocation.reloadwhen a change is detected).src/rendererdirectory. if you make changes to source code in the main process, background processes, or preload scripts, you'll still have to manually stop restart the server to see those changes.src/preloaddirectory. we can choose to bundle/process these but for now, we don't because they work in a different environment (access to both browser and Node apis because Node integration is turned on).staticdirectory. this way we don't deal with configuring loaders for certain assets right now, which was causing some asset duplication.react-intluses for auto id-ing messages based on file path. as a result, the flow for defining messages is identical to how we do it in Mapeo Mobile. i had to update every call todefineMessagesto manually specify something like{ id: ..., defaultMessage: ...}for each message string. Based on the lack of a diff in the generatedmessages/, i think my change has been completed successfully. Just need to keep this in mind moving forward.main.html)windowin themain.jspreload script, which like a partial recommendation for when node integration is enabled (https://github.com/electron/electron/issues/9920#issuecomment-468323625)Known issues
Let it be known that I'm not a bundling wizard. It's very possible that the build configuration I've implemented so far needs major reworking based on how browsers and Electron code work. Some other notable issues include:
externalhave to be imported viarequirewhen used in the main renderer code. for example, most of the node apis are listed as external and can only be used by doing something likeconst path = require('path'). this is not really a blocking issue, but just a notable change that may be slightly inconvenient.Remaining work
pathmodule such as building a file path viapath.join, it should be abstracted to a file located somewhere likesrc/renderer/node/path.jsas a helper function, which is then used by the component. my reasoning for this:builder.config.jsconfig to make sure that changes in output are accounted for in terms of what needs to be included/excluded when building the electron app. i'm sure i've missed some stuff there since i haven't specifically tested or focused on it yet.