Evaluate what would be involved in updating to the current version of d3 (v7)

[thomaswilburn]

This will depend on upgrading the rig to handle ES modules, but probably won't necessitate too much more on top of that.

[thomaswilburn]

The rig has been (theoretically) updated to handle ES modules.

Migration guides from D3: V5-V6: https://observablehq.com/@d3/d3v6-migration-guide V6-V7 (release notes): https://github.com/d3/d3/releases/tag/v7.0.0

It doesn't look to me like there are real syntax/API problems for our graphics. I think we could proceed as follows:

1. Pick some standout graphics, preferably one for each of the major template types, and use those for testing.
2. Update our individual modules used in those graphics (e.g., moving from d3-array@2 to d3-array@3).
3. If individual modules work, upgrade the overall D3 library, which we don't really use anywhere but just in case.

[thomaswilburn]

Unexpected snag! D3 exports itself as ES modules, but still maintains a UMD build in the /dist folder for bundlers. However, while Browserify happily ignores the package.json and loads this file, Node itself does not: the "exports" key in the package.json doesn't mark it as public in a way that Node will follow. You can import in Node, but you can't require.

This is a problem because in our HTML, we sometimes use D3 that's ported from the client-side code to do templating. We need to require() it, because we want modules to resolve from the graphics-js folder, not from the rig location, which is what import does (and this doesn't appear to be monkey-patchable).

We're going to file a bug with D3 and hold off on upgrading for now, but our eventual options look like this:

• If D3 fixes the bug - we're all good, we can require these modules to our hearts' content.

OR if D3 declines to fix the bug:

• we move ahead with the upgrade, but specifically warn against using require() for modules in EJS templates. You can still do basic Node processing for data in a template, but nothing requiring a library. This sucks but is sustainable long-term
• we don't upgrade until D3 changes syntax or introduces significant new features, which they did not really do in the 5->7 transition. This creates a potential stumbling block that I don't know how to circumvent
• we create an additional hook for requiring modules that's added to the localRequire in processHTML.js, piping the file through Babel before passing it to Node This isn't terrible, but adds complexity
• we create an alternate load mechanism that's specifically used for EJS templates, similar to but not the same as require(), that can handle both ES (via await import()) and CJS modules, making sure that the pathing is correct for the graphics folder and not the host app. This is elaborate and probably fragile.

[thomaswilburn]

Perhaps unsurprisingly, the D3 project is not at all interested in fixing this bug, so we'll have to figure out other options. https://github.com/d3/d3/issues/3532

[wemod123]

Totally not understand why d3 package only support import but not require,

@thomaswilburn, thank you for the detail advice, we are creating a separate loading mechanism only to handle d3 package, and making it as kind of function-api, really uncomfortable