[web] ES6 modules

[josephrocca]

Is your feature request related to a problem? Please describe. I'm trying to get ONNX Runtime Web working in Deno. Deno maintains very close compatibility with the browser, but there's no concept of HTML/DOM since it's a server-side language, so the usual script tag import method is not possible.

System information

• ONNX Runtime version (you are using): 1.10.0

Describe the solution you'd like It would be great if the package had an ES6 modules build available so that it could be imported like this:

import ort from "https://cdn.jsdelivr.net/npm/onnxruntime-web@1.10.0/dist/ort.es6.js"

[josephrocca]

I've just realised that you can simply write this: (Edit: wrong - read next comment)

import "https://cdn.jsdelivr.net/npm/onnxruntime-web@1.10.0/dist/ort.js"

And it will create the window.ort global. Not ideal since it pollutes the global namespace, but at least this means it's possible to import it into Deno.

I'll leave this issue open since it still would be nice to have proper ES6 module support.

[josephrocca]

Oh, but now I see that a new problem arises. If we do a bare module import like this:

import "https://cdn.jsdelivr.net/npm/onnxruntime-web@1.10.0/dist/ort.js";

then ort.js isn't able to find the wasm file because document.currentScript.src doesn't exist in modules (and ort.js is treated like a module in this case). So, to fix this, all references to document.currentScript.src should try to fall back to import.meta.url.

Here's an example of some code in ort.js trying to detect where it's running, but I think this code is actually generated by webpack? (I haven't used webpack before)

var _scriptDir = typeof document !== 'undefined' && document.currentScript ? document.currentScript.src : undefined;

You can see other examples by searching onnxruntime-web@1.10.0/dist/ort.js for "currentScript" using Ctrl+F.

I was only able to find one reference to document.currentScript.src within the ONNX Runtime Web codebase itself.

@hanbitmyths, @fs-eire Wondering if you could provide guidance on if/how I can submit a pull request to fall back to import.meta.url when ort.js is trying to determine its own location? Specifically in regard to how this _scriptDir stuff is generated by the bundler. This would allow the library to work in Deno and with a "bare" ES6 import statement in the browser.

Note that currently it falls back to __filename which has a default hard-coded value of /index.js, which causes ort.js to try to load the wasm file from /ort-wasm-simd-threaded.wasm. So, if it's possible (?) to change the default __filename to a runtime-generated value, then a quick solution might be to set it to import.meta.url by default.

[fs-eire]

I will take a look at this issue.

[nestarz]

Any workarounds to make onnx runtime works on Deno ?

[josephrocca]

@fs-eire Wondering if you could just add this to the top of the published ort.js file?

if(typeof document === "undefined") {
  var document = {currentScript:{src:import.meta.url}};
}

I.e. just add a "dummy" document to fix the issue in Deno and other JS runtimes that don't have document/DOM stuff. It's not a "proper" fix, but is simple enough that it might make sense until the webpack stuff mentioned in my previous comment can be sorted out?

[fs-eire]

I found that Emscripten actually supports this feature. I am trying to make a change to support it for ORTWEB.

[nestarz]

I published a repo with a working version of onnx-runtime-web on Deno (note: it discard multithreading !).

https://github.com/nestarz/onnx_runtime

I built the wasm files with:

set_property(TARGET onnxruntime_webassembly APPEND_STRING PROPERTY LINK_FLAGS " -s EXPORT_ES6=1 -s MODULARIZE=1 -s USE_ES6_IMPORT_META=1")

[fs-eire]

@nestarz thank you so much for this example! it's a good learning for me.

Currently, we have 3 different scenarios for consuming the onnxruntime-web library:

1. consume from script tag ( <script src="https://some-cdn-server/ort.min.js">):
2. consume from module import ( import * as ort from 'onnxruntime-web' )
3. consume from module URL (import * as ort from "https://deno.land/x/onnx_runtime/mod.ts";)

There are a few hacks we currently doing in our build process :

• the Emscripten generated file ort-wasm.js and ort-wasm-simd.js are identical in content. To reduce the final file size (ort.min.js), we force it to always use ort-wasm.js. This also apply to ort-wasm-threaded.js vs ort-wasm-simd-threaded.js.

  the standard way:

    import ortWasmFactory from './binding/ort-wasm.js';
    import ortWasmSimdFactory from './binding/ort-wasm-simd.js';
  
    ...
    if (simd) {
        ortWasmSimdFactory()...
    } else {
        ortWasmFactory()...
    }

  onnxruntime-web currently doing:

    import ortWasmFactory from './binding/ort-wasm.js';
  
    ...
    const config = {
       locateFile: (fileName: string, scriptDirectory: string) => {
          ...
          if (fileName == 'ort-wasm.wasm' && simd) {
             return 'ort-wasm-simd.wasm';
          }
       }
    };
    ortWasmFactory(config)...

  This indeed helped to reduce the final javascript size, but it caused the problem that webpack failed to analyze the file dependency of ort-wasm-simd.wasm, so it will not copy the web assembly file and modify the file path as it does by default.

• inline worker for proxy mode. The motivation of doing this is for simple distribution (scenario.1). By using the worker-loader webpack plugin, we are able to put whole content of the worker source code into an inline string so that we can distribute single file (ort.min.js).

  standard way:

    proxyWorker = new Worker('./proxy-worker/main');

  onnxruntime-web currently doing:

    proxyWorker = require('worker-loader?inline=no-fallback!./proxy-worker/main').default() as Worker;

  the require(worker-loader?...) can only be understood by webpack with configured to use worker-loader plugin. Webpack recommended to use its builtin worker loader however it does not support inline mode.

• override processing of /ort-wasm.*\.worker\.js$/ to use asset/source. This also helps to inline the .worker.js file, but instead of using in proxy mode, this file is generated by Emscripten in multi-thread build. The file is processed as "asset" instead of source code, to work with special code so that we can still distribute a single file.

  standard way:

    // no extra code

  onnxruntime-web currently doing:

    const config = {
       locateFile: (fileName: string, scriptDirectory: string) => {
          if (useThreads && fileName.endsWith('.worker.js') && typeof Blob !== 'undefined') {
              return URL.createObjectURL(new Blob(
                  [ require('./binding/ort-wasm-threaded.worker.js') ],
                  {type: 'text/javascript'}));
          }
        }
      }
      ...

• use function.toString() on the exported variable from file ort-wasm-threaded.js to reduce file size. This hack specifically rely on the UMD output type for the generated files and webpack config to preserve symbol name of _scriptDir and ortWasmThreaded to work.

  standard way:

    // no extra code

  onnxruntime-web currently doing:

    const config = {
        ...
    };
  
    if (useThreads) {
      if (typeof Blob === 'undefined') {
        config.mainScriptUrlOrBlob = path.join(__dirname, 'ort-wasm-threaded.js');
      } else {
        const scriptSourceCode = `var ortWasmThreaded=(function(){var _scriptDir;return ${factory.toString()}})();`;
        config.mainScriptUrlOrBlob = new Blob([scriptSourceCode], {type: 'text/javascript'});
      }
    }

combining of the hacks, it makes ORT web very difficult to support ES6 module for now. We need to re-think the balance between the benefit of (single-file distribution and file size) VS. (easy-to-maintain code and webpack friendly)

[fs-eire]

15772: This change turns onnxruntime-common into a ESModule/commonJS "dual" package, which means it is able to comsumed as either a ESM or CJS module.

When this is merged, following-up work will start for onnxruntime-web.

[josephrocca]

Just realised that OR Web currently works with Deno by simply setting ort.env.wasm.wasmPaths to avoid the issue I mentioned in my earlier comment. For example, this works:

import "https://cdn.jsdelivr.net/npm/onnxruntime-web@1.15.0/dist/ort.js";
ort.env.wasm.wasmPaths = "https://cdn.jsdelivr.net/npm/onnxruntime-web@1.15.0/dist/";
let onnxImageSession = await ort.InferenceSession.create("https://huggingface.co/rocca/openai-clip-js/resolve/main/clip-image-vit-32-float32.onnx", { executionProviders: ["wasm"] });

[fs-eire]

added ESM exports for onnxruntime-web in latest main branch. ( will later paste nightly build version )

However ort.env.wasm.wasmPaths may still need to manually set.

Please try it out. If it's still not working, let me know the repro steps and I will fix

[josephrocca]

Awesome! Awaiting nightly script CDN URL 🫡

[fs-eire]

Please try 1.17.0-dev.20231014-ae211999dd

[josephrocca]

Hmm, for some reason on JSDelivr I get a 403 HTTP error with this message: "Package size exceeded the configured limit of 150 MB.":

let ort = await import("https://cdn.jsdelivr.net/npm/onnxruntime-web@1.17.0-dev.20231014-ae211999dd/dist/esm/ort.js");

I tried npm install onnxruntime-web@1.17.0-dev.20231014-ae211999dd and then loaded from node_modules like this:

let ort = await import("./node_modules/onnxruntime-web/dist/esm/ort.js");

Then it loads but then throws:

error: Uncaught (in promise) Error: no available backend found. ERR: [wasm] ReferenceError: __dirname is not defined
    at resolveBackend (file:///pathtofolder/node_modules/onnxruntime-web/dist/esm/ort.js:105:13)
    at async Function.create (file:///pathtofolder/node_modules/onnxruntime-web/dist/esm/ort.js:1025:26)
    at async file:///pathtofolder/main.mjs:35:24

And if I try:

import ort from "./node_modules/onnxruntime-web/dist/esm/ort.js";

Then it throws:

error: Uncaught SyntaxError: The requested module './node_modules/onnxruntime-web/dist/esm/ort.js' does not provide an export named 'default'
import ort from "./node_modules/onnxruntime-web/dist/esm/ort.js";

And using esm.sh like this:

import ort from "https://esm.sh/onnxruntime-web@1.17.0-dev.20231014-ae211999dd/dist/esm/ort.js";

also throws:

error: Uncaught SyntaxError: The requested module 'https://esm.sh/onnxruntime-web@1.17.0-dev.20231014-ae211999dd/dist/esm/ort.js' does not provide an export named 'default'
import ort from "https://esm.sh/onnxruntime-web@1.17.0-dev.20231014-ae211999dd/dist/esm/ort.js";