Build System: Replace Brunch with Modern Tooling

daveseah commented 11 months ago

The build tool brunch that is used to compile the NetCreate app is no longer actively developed or supported. For future-proofing, we may want to consider replacing it with a more modern tool.

BACKGROUND

Brunch is the Javascript Build Tool aka "bundler" that's currently used in NetCreate-2018 and still used in NetCreate-ITEST. At the time it was a contender to Webpack, the bundler that eventually won out. Brunch is no longer actively maintained and many of its dependencies require security updates. However, the true security needs are on the webserver side, not the developer's build tool side; in other words, so long as the web app itself is secure, the tools that we use to build the web app maybe don't have to. This is something we will need to untangle, probably when we get to the server architecture point.

TECH REVIEW

I've made a note of what the current bundler process does:

The brunch command uses brunch-config.js to tell it how to combine our source files into "bundles" of javascript and css to be copied to public.
There are three such bundles defined:
- one that contains all the javascript in any directory beginning with app
- one that contains all the javascript in a directory NOT beginning with app (node_modules)
- all css from anywhere
The config file copies the directories app-data, app-template, htmldemo, and app-config into the top level of the public site
Brunch implicitly copies anything in the assets directory into the top level of public
The web apps "entry point" loads the script bundles, which is defined in app/assets/index.ejs EJS template and delivered by the Express server defined in our custom brunch-server.js. For the static version of the site, an index.html is generated from this template.

SOLUTION

We just need to produce these bundles in an alternate way so they can be included in app/index.ejs. For reference, here's the general structure of that file:

<!DOCTYPE html>
<html>
  <head>
    <title>NetCreate</title>
    <link rel="stylesheet" href="/styles/netc-app.css" type="text/css" />
  </head>
  <body>
    <div id="app-container">
      <div style="padding: 10px;">Loading Net.Create...</div>
    </div>
    <script src="config/netcreate-config.js"></script>
    <script>
      // NC_UNISYS are network parameters
      window.NC_UNISYS = {
        client: {
          ip: '<%=ip%>',
          ukey: '<%=ukey%>'
        },
        server: {
          hostname: '<%=hostname%>',
          ip: '<%=hostip%>',
          ustart: '<%=ustart%>'
        },
        socket: {
          uaddr: '<%=uaddr%>',
          uport: '<%=uport%>'
        }
      };
    </script>
    <script src="/scripts/netc-lib.js"></script>
    <script src="/scripts/netc-app.js"></script>
    <script>
      document.title += ` ${window.NC_CONFIG.dataset} @ ${window.NC_UNISYS.server.ip}`;
      require('init');
    </script>
  </body>
</html>

In the above you can see:

load: CSS styles netc-app.css loaded
load: database config/netcreate-config.js loaded, which sets database params (dynamically generated by nc.js)
set: network identity NC_UNISYS declared
load: support libraries netc-lib.js loaded
load: app code netc-app.js loaded
run: require('init') runs the code from init.jsx, which starts the entire app

Our later projects (e.g. GEMSTEP) already have converted to Webpack so it should not be too much of a problem, just an annoyance to deal with on a scheduled "technical debt" week when we start doing those.

daveseah commented 10 months ago

NetCreate 1.0 Build System Notes

Outlining all the bits of the current brunchjs-based build system to see what we can replace it with.

1. npm package scripts

In package.json we have several scripts defined:

dev -brunch watch -s is the main development version
classroom - brunch build -e classroom was used to generate a non-devserver version for use on a classroom LAN.
package - brunch build -e package is used to create static html demo sites
package:debug and debug are not used in normal practice

2. netcreate config generation

Before running brunch itself, there is a hack to create the app-config/netcreate-config.js file using Ben's nc.js configuration tool, and this is also imported into brunch-config.js as the constant NC_CONFIG

// this file generated by NC command
const NC_CONFIG = {
  dataset: "test",
  port: "3000",
  googlea: "0"
};
if (typeof process === "object") module.exports = NC_CONFIG;
if (typeof window === "object") window.NC_CONFIG = NC_CONFIG;

The nc.js script MUST be run for Netcreate to function, though this is not strongly documented in the codebase itself. At minimum, you can do:

# create app-config/netcreate-config.js to use a database and template
# in the runtime directory 
./nc.js --dataset='test'

# now you can start netcreate
npm run dev

In this example, the files test.loki and test.template.toml are created in the runtime folder. These are used by the appserver to spawn a database instance (LokiJS) and a TOML file that contains the configuration of the "templated features".

3. brunch configuration

Brunch works by collecting all the same types of file into a concatenated bundle stufed into the public directory (brunch default).

The brunch-config.js file is used by the brunch command, and the devserver is loaded in brunch-server.js (these are both hardcoded filenames used by brunch).

The brunch-config.js file specifies these main bundles:

javascript files:
  app* -> scripts/netc-app.js
  !app -> scripts/netc-lib.js
css
  app, node_modules -> styles/netc-app.css

There are specific plugins to handle transpilation and static files.

babel:
  presets: env, react
copycat:
  app-config/*    -> public/config
  app-data/*      -> public/data
  app-templates/* -> public/templates
  app-htmldemos/* -> public/htmldemos

There are specific environmental switches: classroom, package, and package-debug

classroom - This was the "LAN mode" used in the classroom's local network, but isn't currently used. The config turns on optimization, turns off sourceMaps, disables autoReload.
package - This is used to create the "standalone static html demo" versions. It disables debug and optimization features, and creates JSON versions of the dataset and config. It also creates TOML version of the "template" which is used to rename properties in the Node and Edges display.

4. brunch custom server

The brunch custom webserver is used by brunch watch --server which looks for a hardcoded brunch-server.js filename.

Ours does quite a bit in configuration:

Checks for nvm and node version compatibility
starts the UNISYS message network server
config Express for cross-domain cookies
config Express to use EJS template engine
config route / to serve index.ejs with the values from netcreate-config
config all other routes to be handled bu Express.static

When the server is finally started, the callback does a number of additional checks.

report the git branch
report the system architecture, and whether ARM was detected
report the node version, and whether there is a mismatch with nvmrc

Then it configures the underlying system:

Registers all server-side UNISYS message handlers
Starts the UNISYS central message dispatcher

The server instance also gets an error handler to detect EADDRINUSE

daveseah commented 10 months ago

NOTE: The template system is complicated and its code is extremely difficult follow. There are legacy datafiles in app-templates and app-data that may or may not be related. Awaiting confirmation on what these are currently being used for.

netcreateorg / netcreate-itest