Infrastructure: Module Architecture Stage 2

[daveseah]

This stage defines a new modular code system that will be used for NetCreate 2.0 features (e.g. commenting, data processing). The intent is to create a portable system that (1) can be easily added to existing and new projects and (2) maintain the highest possible developer experience (DX) for programmers of all skill levels.

OVERVIEW OF CHANGES

There are now two build systems and three packages.

1. The original brunch 2 build system used with NetCreate for the main root directory that works as it did before.
2. The new URSYS core library using esbuild with its own package.json exporting @ursys/netcreate
3. The add-on module library, also using esbuild with its own package.json, exporting @ursys/nc-addons

The new build system defines multiple "entry points" for each package.json. The new code is exported to _ur/_dist as both legacy CommonJS modules and the newer ES Modules in either nodejs or browser contexts.

The repo is also technically a "monorepo" now, which allows the root-level npm commands to handle multiple independent packages at the same time when using npm ci and npm build similar to lerna in the GEM-STEP project.

BACKGROUND INFO

This pull request (PR) establishes the seeds of systematic modular thinking by providing an initial implementation, which has two main thrusts:

1. Offer explicit definitions of modularity with an additional emphasis on "model-first" approach to code design and its implementation. This will help developers with different backgrounds understand what the code does by discouraging spaghetti coding.

2. Encourage and standardize the use of existing models in industry to help developers understand the conventions around common-but-complex functions in a systematic way. Topics include asynchronous programming, event handling, animation, user interface, data architecture. This will help developers by providing simple models of use for well-established tools, terminology, and practice.

We are at the very beginning of a team-wide initiative to really nail down how we will do modular programming by capturing and systemizing software concepts as they apply to our realtime projects.

This is a long-term initiative, and there will be future stages of modular architecture refinement as we try out our starting system. The system and its components will be based on the experience we've gained since 2014 working on these particular problems together.

NEXT STEPS

• [ ] try out the module system to coauthor comment management system using dataflow principles in peer programming sessions
• [ ] write description of module system...but where?
• [ ] note that there are different features that are not fully working related to "add ons"
• [x] update whimsical diagram

KNOWN ISSUES

• [ ] new ur modules do not show correct source maps because of brunch issues; workaround is to combine _ur_mods with _ur as a single compiled library instead of two.

HOW TO TEST

This PR does not add any new NetCreate features, but does have minimal integration with the existing codebase that we can test. The most important test is Confirm Netcreate Still Works (Test 1); the other tests are optional to perform but I would appreciate that they are tested.

As the build system has been modified, it's important to do a clean install.

TEST 1: CONFIRM NETCREATE STILL WORKS

1. pull branch dev-ds/ur-modules2
2. open the project in VSCODE through the netcreate-itest.code-workspace project file
3. open the integrated VSCODE terminal. test that NVM detection is working in the shell prompt. It should have the text zshrc: VISUAL STUDIO CODE INTEGRATED TERMINAL DETECTED with possibly different text below. Capture a screenshot of the entire terminal window and post as a comment here (even if successful).

Before proceeding, we want to save your settings just in case...

5. In the same terminal: npm run clean then npm ci
6. backup your runtime directory (this shouldn't be necessary but play it safe)
7. note the name of the dataset that you're currently using by looking at the netcreate-config.js file in the app-config directory.
8. rename the netcreate-config.js to old-config.js or delete the file; you will recreate it in a future step.

Now we run NetCreate

9. Run npm ci followed by npm run dev
10. You should see a warning that informs you that you have not set up NetCreate.
11. Using the dataset you noted from step 6, follow the instructions to run ./nc.js
12. The system should start as normal, recreating the netcreate-config.js file.
13. You may see a lot of warnings during build; this is normal because NetCreate uses very old libraries in its build system but the affected packages are limited to the development environment. For a list of these packages, see #75
14. Make sure that everything looks like it's working.
15. Capture the information you see just before --- compilation complete ---...it should look like this (even on M1 systems):

TEST 2: EXPLORE THE NEW SYSTEM

16. Back at the integrated terminal, cd _ur. This is the home of the secondary build system.
17. Check to see that there is no _dist directory in the folder. it should be grayed out because it's an ignored file.
18. Type ./ur to execute the build system script placeholder.
19. build messages should appear.
20. check the _ur/_dist folder. It should now exist and include a number of .js and .map files.

Now we will test integration with the old NetCreate codebase.

21. Switch back to the main repo directory and run NetCreate
22. Browse to localhost:3000 in a Chromium-based browser (e.g. Chrome or Arc)
23. Open the Javascript console, then click the init.jsx entry to see the full debug context.

Confirm this is what you see

24. There are two colored rectangles labeled UR and URMOD.
25. Note that the @client.ts filename is showing on the right. Click it.
26. Confirm that you are seeing TypeScript source code. On line 20, the term :void appears which is not regular Javascript.
27. There should be no errors in the console, though you may see a single warning about 'force reloading'

TEST 3: CONFIRM LINTING AND TYPE CHECKING

These are the systems that ensure developers using Visual Studio Code are not introduce issues into the code base.

28. Back in the integrated terminal, type npm run lint
29. You'll see a bunch of warnings in the old code base
30. command-click on the first warning in the terminal window. This will take you directly to the file that shows the problem. We usually have this turned off for warnings like no-unused-vars which are the majority of the warnings
31. Find the errors in nc-logic.js. These are the critical errors that we don't want in our codebase.

Now shift attention back to the Visual Studio Code editor itself

32. Open the file app/system/datastore.js (a plain javascript file)

33. On line 26 insert the line 'banana'...red squiggles should appear under the word, indicating an error. Hovering over the highlighted word tells you what the error is.

34. Modify the line to read banana = banana ++ 1. the error shifts. When there are multiple errors in a file, some errors are masked until you fix them.

35. This shows that the ESLINT vscode extension is functioning properly

36. Open the file _ur/browser-client/env-browser.ts (a typescript file)

37. On line 9 insert type FOO = string; followed by const test: FOO = 123;

38. You should see test underlined in red. Hovering over the highlighted area for a second should show you ts errors as well as eslint errors.

39. This shows that Typescript configuration is correctly installed.

40. Open the file _ur/node-server/files.mts (a nodejs typescript module file)

41. Confirm that the type FOO = string; const test: FOO = 123; generates the same error as the previous version.

TEST 4: HIDDEN FILES

There are several files that are now excluded from the file explorer. These files are generally things that no one should be touching, not even other developers. Here's how to find them, though:

42. Open VSCODE Settings
43. Click the workspace tab to show the settings saved in netcreate-itest.code-workspace
44. Type "Files: Exclude" into the search settings bar to narrow-down the search.
45. You should see files like .editorconfig in the list:
46. Alternatively you can open the netcreate-itest.code-workspace file directly, which should have this section:

---

That concludes the testing of surface features in this pull request.

[daveseah]

TECH NOTES

These new modules are intended to function in at three contexts from a single set of source files:

1. as an import into NetCreate to add new features
2. (optional) as a batch processing module running in the CLI
3. (optional) as a participant in URSYS lifecycle events (e.g. URNET)

Initial attempt to update the build system proved too difficult due to the legacy nature of the NetCreate 1.0 codebase and could have taken months. The next course of action was to add a secondary build system alongside the old such that new code to still run inside the old code without ruining the developer experience. This also helps with the plan to make our modules reusable in the future.

see source Whimsical Diagram for updates

Outline of Changes

• fix NetCreate sources that didn't adhere to CommonJS module format
• modify root package.json to use multiple workspaces, add package.json files for new workspaces, modify build scripts to use workspaces
• add global type declarations for NodeJS and other packages
• make CLI tool to run file-based build tasks that live in new workspace directories
• add typescript, eslint, prettier configuration for VSCODE live linting suppport
• add esbuild task to compile the @ursys/netcreate core library into CJS, MJS, JS with conventions to keep the files separate from each other using either TSNode Typescript or Node CJS modules.
• add second layer of esbuild tasks to create "modules" and "addons" that can freely import @ursys/netcreate
• clean-up obsolete data files, declutter VSCODE file tree by hiding rarely-edited files (e.g. .eslintrc.js), improve prompts for detected NodeJS version and netcreate-config.js status
• test import of @ursys/netcreate into old codebase, ensuring that source maps correctly survive the conversion from ESM to CJS modules (needed for compatibility) in the source Typescript language
• modify main dev task so it also runs the build tasks needed for @ursys/netcreate

Dropped features:

• multi access modules for cll, message, and direct import (skeleton is there, but not implemented because we don't know the actual needs yet)

[benloh]

Reviewing...

TEST 1: CONFIRM NETCREATE STILL WORKS

1. Confirm nvm use

2. Confirm everything works...Template Edit BUG (Ben's bug probably) -- To reproduce

   • Go to Edit Tempalte
   • Click on "Edit Node Types"
   • Click "Add row" Uncaught ReferenceErorr: f is not defined. Template.jsx:146

TEST 2: EXPLORE THE NEW SYSTEM

16. Type ./ur -- results in an error:

[daveseah]

Oh, that is inadvertently including some test code that I didn't think I committed yet. Good catch!

[benloh]

Confirmed ./ur error fixed.

All other tests passed.

[daveseah]

I've pushed a few refinements to detect zsh vs bash issues, but I don't think you need to retest. I've updated the testing instructions also to fill-in missing information.

[benloh]

Looks like the Template.jsx fix also worked. Thanks! Merge away?

[daveseah]

Looks like the Template.jsx fix also worked. Thanks! Merge away?

@benloh I did one final change: removed the restriction on requiring zsh, which also opened up compatible detection for linux. This means that it should work with bash OR zsh

If you could give the PR one last check to see if npm run dev works both inside VSCODE integrated terminal and also in a standalone terminal window. The output should be different in each case, but really just wanting to avoid a crash

[daveseah]

@benloh You don't have to retest anything else, just run npm run dev to make sure it doesn't crash.

[daveseah]

Updated Script for Environment Detection

Examples of the output you should be seeing (these are zsh on macos on an M1):

Running inside VSCODE opened through the provided netcreate-itest.code-workspace (these have been standard in our projects for quite some time):

Running inside VSCODE but folder was opened directly, skipping the workspace project which has project-specific settings, in particular ensuring that certain plugins are turned off and running workspace scripts:

Running outside of VSCODE in a regular terminal window

The new script should on bash now as well as I am not using any (?) zsh features. Also, this should cover the case of running linux on DO as well but I haven't tested that.

[daveseah]

There is now also a hard stop if there is a node version mismatch. This has been there for a while but thought I would point it out:

[benloh]

Confirmed:

• I'm getting CODE-WORKSPACE messages in VSCode and non-VSCode runs.
• Getting NODE VERSION MISMATCH warnings.
• bash and zsh both work Merging!