[Feature]: support disable scope hoist and inject custom module runtime

[hardfist]

We wrap esbuild as our build tools and want to support HMR, CSS HMR is easy, but when it comes to js HMR, the main obstacle is scope hoisting will remove the module info in the bundle file, and if esbuild could supports disable scope hoisting in development mode and supports custom module runtime, it maybe possible for upper level tools to support HMR, which is every important for large scale application.

[hardfist]

rollup could support HMR by custom plugin https://github.com/rixo/rollup-plugin-hot , which use rollup's preserveModules( which is the way rollup disable scope hoisting)

[evanw]

You haven't provided an example of the code transformation you want to apply so I don't understand what you're saying. But in case it helps: If you want to swap out the exports of some module for some other module, you could consider making sure the module is in CommonJS format instead of having it be in ESM format. Exports in CommonJS modules are dynamic (i.e. can change at run-time) while exports in ES modules are static (i.e. are fixed at run-time), which seems like a better fit for what you're doing. You should be able to change the properties of the CommonJS exports object to swap in another module. This could either be done by mutating the object or by setting the object to a proxy that then changes where the exports come from.

[hyrious]

HMR is not only about module replacement, but also refreshing side-effect. You can not know if a previous module holds any value that can not be replaced by your HMR runtime. For example, const { x } = require('lib'), then he always holds previous x even if you swapped the lib.

The basic idea of rixo's plugin and vite and webpack's hmr design is let some file act as a boundary to refresh its effects (and its dependencies' effects), sort of react's useEffect but in modules context (regardless of the module is esm (vite) or cjs (webpack) of systemjs (rixo)). The refreshing is quite simple: import('same-file?timestamp').

So the basic problem is: how to track files' dependency graph and find boundary files to reload themself?

Vite's answer is: At the server side it uses es-module-lexer and some string matching function to scan files to build the module graph (moduleGraph.ts), the ones with import.meta.hot becomes boundary, everytime his file watcher found some change, it will find the correct boundary and send update events to the browser (hmr.ts).

rixo's answer is: Someone's fork of SystemJS adds the reload feature that takes care of all things mentioned above. Besides, SystemJS can build the module graph at the client side by adding hooks. So he transforms all your code to SystemJS and use that feature to achieve HMR.

webpack -- I don't know, I never used it.

[hardfist]

@hyrious @evanw To fully support HMR, we basic need three level supports

• module loader supports replace module at runtime (bundler support disable scope hoist and keep module info in bundler's result in development mode, webpack supports by custom module loader(webpack.require) and rollup supports this by preserveModule which keeps module seperately file in output, vite supports this by bundleness(module is untouched), esbuild partially supports this by override commonjs runtime(__toCommonjs function))
• module graph support analysis changed effect module boundary: analysis the effected ancestor module boundary
• framework runtime support (like react fast-refresh) The last two actually is not so related to bundler(which can be built on top of any boundler), but the first one need bundler supports(which is what I hope esbuild could support), even override commonjs may work, it's not an official api and try to convert every es module to commonjs may introduce extra cost

I list webpack | esbuild | rollup's different generated result to supports hmr in https://github.com/hardfist/esbuild-hmr

[hardfist]

I also check bun's HMR strategy，it seems bun also support something like preserveModule in development mode, even though bun not supporting custom HMR logic, but it supports react HMR very well, I'm wondering whether esbuild could support some thing like this. @Jarred-Sumner may have better idea?

• input

  // index.js
  import { answer } from './lib'
  setInterval(() => {
  console.log('answer:',answer);
  },1000)

  // lib.js
  export const answer = 42;

• output

  
  //index.js
  import {
  __FastRefreshModule as FastHMR
  } from "http://localhost:3000/node_modules.3d76e15706e74636.bun";
  import {
  __HMRClient as Bun
  } from "http://localhost:3000/node_modules.3d76e15706e74636.bun";
  import {$2d42243f} from "http://localhost:3000/node_modules.3d76e15706e74636.bun";
  var FastRefresh = $2d42243f();
  import {answer} from "http://localhost:3000/src/lib.js";
  Bun.activate(false);

var hmr = new FastHMR(3465849059, "src/index.jsx", FastRefresh), exports = hmr.exports; (hmr._load = function() { console.log("answer:", answer); hmr.exportAll({}); })(); hmr._update = function(exports) { };

```js
//lib.js
import {
__FastRefreshModule as FastHMR
} from "http://localhost:3000/node_modules.3d76e15706e74636.bun";
import {
__HMRClient as Bun
} from "http://localhost:3000/node_modules.3d76e15706e74636.bun";
import {$2d42243f} from "http://localhost:3000/node_modules.3d76e15706e74636.bun";
var FastRefresh = $2d42243f();
Bun.activate(false);

var hmr = new FastHMR(4213202960, "src/lib.js", FastRefresh), exports = hmr.exports;
(hmr._load = function() {
  var answer = 45;
  if (import.meta.hot)
    console.log("met hot:");
  hmr.exportAll({
    answer: () => answer
  });
})();
var $$hmr_answer = hmr.exports.answer;
hmr._update = function(exports) {
  $$hmr_answer = exports.answer;
};

export {
  $$hmr_answer as answer
};

[hyrious]

@hardfist I just tried implementing HMR in esbuild, and it turns out that module-replacement isn't really needed (your first requirement) when using the vite style HMR, check it here: https://github.com/hyrious/gua/blob/develop/scripts/hmr.ts (don't care the other contents outside of scripts folder :p).

I borrowed some ideas from vite and implemented this thing in only about 200+ lines (without many options or features, just give it a try). I also copied their prefresh (the HMR lib for preact) plugin, which only depends on one api -- hot.accept(), and it worked well. So I guess it proves that it is possible to make HMR work in esbuild (serve mode).

Your ideas about module-replacement is easy to think of, but it actually will make things harder -- like what I mentioned above, you cannot prevent people from destructing the exported object (maybe a bundler can, but I don't think it will work well in all situations), including nested ones.

One problem with current design is, it serves first response with everything bundled (by esbuild serve). Let's say we have a file has some global state: export let state = { count: 1 }. When HMR refreshing happens, it will execute with another state bundled. I guess that's why vite chooses no-bundle with this design, in which case the browser is still able to use the same module.

[Jarred-Sumner]

I'm wondering whether esbuild could support some thing like this. @Jarred-Sumner may have better idea?

It is difficult to be unopinioniated when it comes to HMR. The code transformations part is simple. It's everything else that makes it complicated.

This is what bun does

Most of the work for Bun's HMR happens client-side in hmr.ts. This keeps the server mostly stateless (watching is inherently stateful), which is important for performance

When a JavaScript-like or CSS file previously requested changes, bun sends a WebsocketMessageFileChangeNotification over the websocket:

export interface WebsocketMessage {
  timestamp: uint32;
  kind: WebsocketMessageKind;
}

export interface WebsocketMessageFileChangeNotification {
  id: uint32;
  loader: Loader;
}

Then, the client looks at the list of loaded modules and tries to find a matching module ID. If it can, then it sends a WebsocketCommandBuild

export interface WebsocketCommandBuild {
  id: uint32;
}

Eventually (usually in < 2ms), the server responds with one of these

export interface WebsocketMessageBuildSuccess {
  id: uint32;
  from_timestamp: uint32;
  loader: Loader;
  module_path: string;
  blob_length: uint32;
}

export interface WebsocketMessageBuildFailure {
  id: uint32;
  from_timestamp: uint32;
  loader: Loader;
  module_path: string;
  log: Log;
}

// This is bun saying "I don't know what that file ID means, can you tell me?"
export interface WebsocketMessageResolveID {
  id: uint32;
}

From there, if it built successfully, we fork() the dependency graph

const oldGraphUsed = HMRModule.dependencies.graph_used;
var oldModule = HMRModule.dependencies.modules[this.module_index];
HMRModule.dependencies = orig_deps.fork(this.module_index);

// earlier
 class DependencyGraph {
    modules: AnyHMRModule[];
    graph: Uint32Array;
    graph_used = 0;

    fork(offset: number) {
      const graph = new DependencyGraph();
      graph.modules = this.modules.slice();
      graph.graph_used = offset;
      graph.graph = this.graph.slice();
      return graph;
    }
  }

And then import the new module:

var blobURL = null;
try {
  const blob = new Blob([this.bytes], { type: "text/javascript" });
  blobURL = URL.createObjectURL(blob);
  await import(blobURL);
  this.timings.import = performance.now() - importStart;
} catch (exception) {
  HMRModule.dependencies = orig_deps;
  URL.revokeObjectURL(blobURL);
  // Ensure we don't keep the bytes around longer than necessary
  this.bytes = null;

  if ("__BunRenderHMRError" in globalThis) {
    globalThis.__BunRenderHMRError(
      exception,
      oldModule.file_path,
      oldModule.id
    );
  }

  oldModule = null;
  throw exception;
}

With ESM, export names for a module cannot change. That means we have to keep a reference to the module originally exported at a URL, but we usually only need to keep the first one.

const isOldModuleDead =
oldModule &&
oldModule.previousVersion &&
oldModule.previousVersion.id === oldModule.id &&
oldModule.hasSameExports(oldModule.previousVersion);

if (!isOldModuleDead) {
  HMRModule.dependencies.modules[this.module_index].additional_updaters.push(
    oldModule.update.bind(oldModule)
  );
  HMRModule.dependencies.modules[this.module_index].previousVersion = oldModule;
} else {
  HMRModule.dependencies.modules[this.module_index].previousVersion =
    oldModule.previousVersion;
  HMRModule.dependencies.modules[this.module_index].additional_updaters =
    origUpdaters;
}

^{The mix of camelCase and underscore_case is because Zig uses underscore and JS camelCase and I constantly forget}

From there, we try to find a module which doesn't export a React component (specific behavior expected by React Fast Refresh) and replace the exported bindings by calling mod.update()

for (
  let i = 0;
  i <= end;
  i++
) {
  const mod = HMRModule.dependencies.modules[i];
  if (!mod) continue; // this array is holey sometimes
  let handled = false;

  if (!mod.exports.__hmrDisable) {
    if (typeof mod.dispose === "function") {
      mod.dispose();
      handled = true;
    }
    if (typeof mod.accept === "function") {
      mod.accept();
      handled = true;
    }

    // If we don't find a boundary, we will need to do a full page load
    if ((mod as FastRefreshModule).isRefreshBoundary) {
      foundBoundary = true;
    }

    // Automatically re-initialize the dependency
    if (!handled) {
      mod.update();
    }
  }
}

After all that is over, we tell React, "i'm ready for you to refresh"

if (pendingUpdateCount === currentPendingUpdateCount && foundBoundary) {
  FastRefreshLoader.RefreshRuntime.performReactRefresh();
  // Remove potential memory leak
  if (isOldModuleDead) oldModule.previousVersion = null;
} else if (pendingUpdateCount === currentPendingUpdateCount) {
  FastRefreshLoader.performFullRefresh();
} else {
  return Promise.reject(
    new ThrottleModuleUpdateError(
      `Expected pendingUpdateCount: ${currentPendingUpdateCount} but received: ${pendingUpdateCount}`
    )
  );
}

[hyrious]

To overcome the problem I said before, I found that we still need something doing replace module at runtime. But it's not only about replace, it's used to keep dependent modules references. Vite supports this in native ESM way by using the browser cache of the requests from import statements. Yeah their cache is not only used for better performance, but also for the correctness of HMR.

So sadly speaking, if we want to make the vite/rollup/webpack HMR spec work in esbuild, there're only 2 choices:

• Native ESM, just go Vite (to no-bundle, otherwise it won't work).
• CommonJS, maybe esbuild could let us hack into the module graph. The HMR works as follows:
  1. Files with hot.accept() are boundaries.
  2. When some files change, find the common boundary file of them.
  3. Execute all related files' hot.dispose() callback.
  4. Download and put the changed files' require_lib callback in the module graph but don't call them.
  5. Execute the boundary file's require_xxx(), recursively triggers these require_lib() calls.
  6. Run the boundary file's hot.accept() after 5, if defined.

[hardfist]

@hyrious I try to run your example, but failed

[hyrious]

@hardfist It works fine both on my Windows 10 and macOS 12.2.1, with npm or pnpm. Sorry but maybe you can give me some instructions on how to reproduce it.

[hardfist]

I found the problem,I click the hmr link so it crash, but it seems that what you implement is not HMR, it doesn't supports Module replacement, it only supports bundle replacement, which is not useful. @hyrious

[hyrious]

@hardfist Yeah I agree with you, the main problem is the runtime (regardless of it is in node or in browser) need to manipulate the module graph. Vite supports this in native ESM way where it must be no-bundled. Other choices can only hack into the require calls.

[hardfist]

that's what i said

• bundleness like vite
• preserveModule like rollup
• custom require like webpack

preserveModule seems more sutible for esbuild, since it's just an strategy of code splitting.

[hardfist]

@evanw @hyrious we successfully implement real HMR ourself on top of esbuild, but it also carries some performance problems we need to tackle. we implement HMR by

• we build module graph by onResolve&onLoad hook, which is nearly the same as webpack's dependency graph
• we transform every module into our own module system like webpack.require
• we generate the finally code(by module graph) like webpack's HMR result without using esbuild's generated result

but it comes with some cost

• in the module graph we need the resolved moduleId path, which esbuild doesn't expose, so we use the build.Resolve on resolve hook to get the resolvedPath, which has some significant cost
• we transform file to our own module system which has some cost
• we package module graph to bundle file which also has some cost

so I'm wondering whether esbuild could

• expose esbuild's internal moduleGraph info to plugin hook
• optimize build.onResolve performance(which's slow because it runs every plugin)

Or will esbuild accept MR to intergrate some HMR related feature to esbuild.

It seems remix also has some problems about HMR support, @ryanflorence @mjackson may be interested