Can't import Typst.ts module in Node.js

[millianlmx]

Describe the bug I try to import the typst.ts module using the following code :

import { createTypstRenderer } from '@myriaddreamin/typst.ts';
const renderer = createTypstRenderer();

renderer.init();

And I got the following error :

node:internal/modules/cjs/loader:1042
  throw err;
  ^

Error: Cannot find module '@myriaddreamin/typst-ts-renderer'
Require stack:
- /home/millian/test/node_modules/@myriaddreamin/typst.ts/dist/cjs/renderer.js
- /home/millian/test/node_modules/@myriaddreamin/typst.ts/dist/cjs/index.js
- /home/millian/test/main.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1039:15)
    at Module._load (node:internal/modules/cjs/loader:885:27)
    at Module.require (node:internal/modules/cjs/loader:1105:19)
    at require (node:internal/modules/cjs/helpers:103:18)
    at Object.<anonymous> (/home/millian/test/node_modules/@myriaddreamin/typst.ts/dist/cjs/renderer.js:5:29)
    at Module._compile (node:internal/modules/cjs/loader:1218:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1272:10)
    at Module.load (node:internal/modules/cjs/loader:1081:32)
    at Module._load (node:internal/modules/cjs/loader:922:12)
    at Module.require (node:internal/modules/cjs/loader:1105:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/millian/test/node_modules/@myriaddreamin/typst.ts/dist/cjs/renderer.js',
    '/home/millian/test/node_modules/@myriaddreamin/typst.ts/dist/cjs/index.js',
    '/home/millian/test/main.js'
  ]
}

Node.js v18.13.0

Expected behavior Import and use the module correctly.

Desktop :

• OS: Ubuntu 23.04
• Node.js v18.13.0

Additional context I'm completely new in TypeScript development. There are many things I don't understand and master. Mainly I would like to import the module in Node.js and use it with electron or in my browser directly ? How I suppose to do that ? Thanks in advance. Have a nice day.

[Myriad-Dreamin]

Please install the renderer package by your package manager:

// via npm
npm install @myriaddreamin/typst-ts-renderer
// or yarn
yarn install @myriaddreamin/typst-ts-renderer

When you have any other problem, please let me know 😃.

[millianlmx]

Thank you for your answer. I already installed the @myriaddreamin/typst-ts-renderer package. See my node_modules folder. The error I have given above is with the package installed. So that's why I don't understand the error. Thank you a lot for your help :)

[Myriad-Dreamin]

Thank you for your answer. I already installed the @myriaddreamin/typst-ts-renderer package. See my node_modules folder. The error I have given above is with the package installed. So that's why I don't understand the error. Thank you a lot for your help :)

Oh, I miss it. Could you change the module configuration in your tsconfig.json?

{
  "compilerOptions": {
      "module": "ESNext",
  }
}

You are importing it in commonjs mode.

[millianlmx]

I'm always getting the same error, here is my tsconfig.json file :

{
  "compilerOptions": {
    "rootDir": "src",
    "outDir": "dist/",
    "module": "ESNext",
    "moduleResolution": "node",
    "declaration": true,
    "declarationMap": true,
    "inlineSources": true,
    "inlineSourceMap": true,
    "types": [
      "web"
    ],
    "target": "ES2020",
    "allowJs": true,
    "noImplicitAny": true,
    "importHelpers": true,
    "strictNullChecks": true,
    "stripInternal": true,
    "lib": [
      "ES5",
      "ES6",
      "ES7",
      "ES2018"
    ]
  },
  "include": [
    "src/**/*.ts"
  ]
}

and my package.json :

{
  "name": "typstdrawingtool",
  "version": "0.0.1",
  "description": "An Electron based tool to make typst drawing faster.",
  "main": "main.js",
  "type": "module",
  "scripts": {
    "start": "tsc && node dist/main.js"
  },
  "author": "millian59192@gmail.com",
  "license": "ISC",
  "dependencies": {
    "@myriaddreamin/typst-ts-renderer": "^0.4.0-rc5",
    "@myriaddreamin/typst.ts": "^0.4.0-rc5",
    "@types/web": "^0.0.115"
  }
}

PS: there are many things in my tsconfig file that I copy from this repo. Thanks a lot for your help :)

[Myriad-Dreamin]

@millianlmx I can try to upload a starter template in packages. Please wait a minute.

[millianlmx]

Seriously, it would be great ! Awesome, thank you for help and for your lib :)

[Myriad-Dreamin]

Sorry that I'm late. The node.js can work with typst.ts v0.4.0-rc5 but that's limited and really cursed, so I have adapted the typst.ts library to all of the targets. Please upgrade the library to 0.4.0-rc6 and rerun the renderer example, it should be robust now!

I also update five node.js templates for you, please also check the most suitable one () :D:

node.js-compiler-next
node.js-next
node.js
ts-node-next
ts-node

Recommend: node.js-compiler-next or node.js

[Myriad-Dreamin]

@millianlmx Specific to run typst.ts in electron, you may only instantiate the compiler module at the main thread and only instantiate the renderer module at the renderer thread. Then you pass the artifact data from the main thread and to the renderer thread.

Alternatively, you may also don't have to run a compiler module in node.js but spawn a native process for the editing service like typst-preview.

[millianlmx]

@Myriad-Dreamin Thank you a lot for your huge help. I'm considering to just call the CLI instead of running web compiler. I have to benchmark the web compiler and CLI.

Moreover, I have a question, when calling the compiler we must add source, with file name and typst code, like the following code :

compiler.addSource('/doc.typ', readFileSync(path.join(__dirname, 'doc.typ'), 'utf-8'));
let artifact:Uint8Array =  await compiler.compile({ mainFilePath: '/doc.typ' });

So the question is, I cannot just make :

let artifact:Uint8Array =  await compiler.compile(myTypstCode);

Thank you in advance.

[Myriad-Dreamin]

@millianlmx Hi, the web compiler is actually not for node.js environment, but for some in browser. I could add a rust binding library with neon, but this is not in plan in short term.

I have to benchmark the web compiler and CLI.

It will be appreciated if you could post some sights to figure out the performance bottleneck. I have not do benchmark to the web compiler.

await compiler.compile(myTypstCode);

If you do that, the source code myTypstCode won't have a path for accessing. therefore a dummy entry path is needed, for example /math.typ or /α.typ by default. I'm unsure whether it is well-defined. I will think of that a bit.

[millianlmx]

Hi @Myriad-Dreamin, indeed, when I do :

await compiler.compile(myTypstCode);

I got an error. But I find a workaround, I write the code to a temp file and everything is good.

In fact, I aim to do a GUI tool to make CeTZ drawing faster. So I have to render the code in the browser. I choose node.js and Electron because I don't know how to compile and brig the library directly in an HTML page, or how to bundle everything in a .js file and just import it using a script tag.

Moreover, when I use the web compiler, i c'ant import package like CeTZ. I have a package not found error.

For your request I don't have data to argue, but when I do real-time compiling their no lag, it's magic.

[Myriad-Dreamin]

@millianlmx

Moreover, when I use the web compiler, i c'ant import package like CeTZ.

Unfortunately 😢, The package functionality has been replaced by some stub implementation, since it introduces network operations. As what I said, the web compiler in browser is limited somewhere. Now as users of the web compiler increased, I will survey it and support it soonly.

[millianlmx]

No problem @Myriad-Dreamin, I will stay tune. Thanks you for all of your work. Without you and your lib my basic Idea to make a typstDrawingTool would have been much more difficult. For now I can import the lib in Node.JS so we can close the issue. For the solution, download the provided node.js template. Thank you again 😁

[Myriad-Dreamin]

@millianlmx I have been supported packages and it gets merged. Documentation and v0.4.0 publish will come in some days. But you could give a try with v0.4.0-rc7.

I'm working hard with Documentation for final v0.4.0 publish.

Here is an example:

const accessModel = new window.TypstCompileModule.MemoryAccessModel();
compilerPlugin
  .init({
    beforeBuild: [
      window.TypstCompileModule.withAccessModel(accessModel),
      window.TypstCompileModule.withPackageRegistry(
        new window.TypstCompileModule.FetchPackageRegistry(accessModel),
      ),
    ],
    getModule: () =>
      '/base/node_modules/@myriaddreamin/typst-ts-web-compiler/pkg/typst_ts_web_compiler_bg.wasm',
  })

A customized package registry requires to implement following interface:

export interface PackageRegistry {
  resolve(path: PackageSpec, context: PackageResolveContext): string | undefined;
}

Example:

export class FetchPackageRegistry implements PackageRegistry {
  cache: Map<string, () => string | undefined> = new Map();

  constructor(private am: WritableAccessModel) {}

  resolvePath(path: PackageSpec): string {
    return `https://packages.typst.org/preview/${path.name}-${path.version}.tar.gz`;
  }

  pullPackageData(path: PackageSpec): Uint8Array | undefined {
    ...
  }

  resolve(spec: PackageSpec, context: PackageResolveContext): string | undefined {
    if (spec.namespace !== 'preview') {
      return undefined;
    }

    /// Check cache
    const path = this.resolvePath(spec);
    if (this.cache.has(path)) {
      return this.cache.get(path)!();
    }

    /// Fetch data
    const data = this.pullPackageData(spec);
    if (!data) {
      return undefined;
    }

    /// Extract package bundle to the underlying access model `this.am`
    const previewDir = `/@memory/fetch/packages/preview/${spec.namespace}/${spec.name}/${spec.version}`;
    const entries: [string, Uint8Array, Date][] = [];
    context.untar(data, (path: string, data: Uint8Array, mtime: number) => {
      entries.push([previewDir + '/' + path, data, new Date(mtime)]);
    });
    const cacheClosure = () => {
      for (const [path, data, mtime] of entries) {
        this.am.insertFile(path, data, mtime);
      }

      /// Return the resolved directory to the package
      /// It is then used to access the package data by the access model `this.am`
      return previewDir;
    };
    this.cache.set(path, cacheClosure);

    /// Trigger write out
    return cacheClosure();
  }
}

[millianlmx]

Hello @Myriad-Dreamin, when i run the following code :

import { createTypstCompiler, createTypstRenderer, FetchPackageRegistry, TypstCompiler } from '@myriaddreamin/typst.ts';
import { MemoryAccessModel } from '@myriaddreamin/typst.ts/dist/esm/fs/memory.mjs';
import { withAccessModel, withPackageRegistry } from '@myriaddreamin/typst.ts/dist/esm/options.init.mjs';

const compiler = createTypstCompiler();
  const accessModel = new MemoryAccessModel();
  await compiler.init({
    beforeBuild: [
      withAccessModel(accessModel),
      withPackageRegistry(
        new FetchPackageRegistry(accessModel),
      ),
    ],
    getModule: () =>
      '../../node_modules/@myriaddreamin/typst-ts-web-compiler/pkg/typst_ts_web_compiler_bg.wasm',
  });

I have the following error :

A JavaScript error occurred in the main process
Uncaught Exception:
Error [ERR_REQUIRE_ESM]: require() of ES Module /home/millian/node.js/node_modules/@myriaddreamin/typst.ts/dist/esm/fs/memory.mjs not supported.
Instead change the require of /home/millian/node.js/node_modules/@myriaddreamin/typst.ts/dist/esm/fs/memory.mjs to a dynamic import() which is available in all CommonJS modules.
    at Function._load (node:electron/js2c/asar_bundle:2:13377)
    at Object.<anonymous> (/home/millian/node.js/dist/esm/main.js:5:22)
    at Function._load (node:electron/js2c/asar_bundle:2:13377)
    at Object.<anonymous> (/home/millian/node.js/dist/esm/app.js:5:16)
    at Function._load (node:electron/js2c/asar_bundle:2:13377)
    at loadApplicationPackage (/usr/local/lib/node_modules/electron/dist/resources/default_app.asar/main.js:121:16)
    at Object.<anonymous> (/usr/local/lib/node_modules/electron/dist/resources/default_app.asar/main.js:233:9)
    at Function._load (node:electron/js2c/asar_bundle:2:13377)
    at node:electron/js2c/browser_init:2:122552
    at node:electron/js2c/browser_init:2:122755
    at node:electron/js2c/browser_init:2:122759
    at Function._load (node:electron/js2c/asar_bundle:2:13377)

Do you know how to figure it out, please ? Thank you for your hard work :)

[Myriad-Dreamin]

@millianlmx There are some simple adaption to make them work with node. Perhaps I should update the node template for you :). I will ping you after the template updated.

[millianlmx]

Oups thank you @Myriad-Dreamin for your help, I'm new in node.js and typescript development, so I don't know how to solve this basic issue...

[millianlmx]

Hello @Myriad-Dreamin I successfully the following code :

import * as _1 from '@myriaddreamin/typst-ts-renderer';

import { createTypstCompiler, createTypstRenderer, FetchPackageRegistry } from '@myriaddreamin/typst.ts';
import { MemoryAccessModel } from '@myriaddreamin/typst.ts/dist/cjs/fs/memory.cjs';
import { withAccessModel, withPackageRegistry } from '@myriaddreamin/typst.ts/dist/cjs/options.init.cjs';
import path = require('path');

async function main(coordinate: { x: number, y: number }) {

  let typstCode: string = `#import "@preview/cetz:0.1.2"
#set page(margin: (top: 0pt, bottom: 0pt, left: 0pt, right: 0pt))
#cetz.canvas({
  import cetz.draw: *

  grid((0, 0), (${coordinate.x}, ${coordinate.y}), step: 1, stroke: gray + 2pt)
})`;

  const compiler = createTypstCompiler();
  const accessModel = new MemoryAccessModel();
  await compiler.init({
    beforeBuild: [
      withAccessModel(accessModel),
      withPackageRegistry(
        new FetchPackageRegistry(accessModel),
      ),
    ],
    getModule: () => undefined,
  });

  console.log(path.join(__dirname + "./doc.typ"));
  compiler.addSource(path.join(__dirname + "./doc.typ"), typstCode);
  let artifact: Uint8Array = await compiler.compile({ mainFilePath: path.join(__dirname + "./doc.typ"), format: 'vector' });

  const renderer = createTypstRenderer();
  await renderer.init();

  const svg = await renderer.runWithSession(async session => {
    renderer.manipulateData({
      renderSession: session,
      action: 'reset',
      data: artifact,
    });
    return renderer.renderSvgDiff({
      renderSession: session,
    });
  });

  return svg;
}

main({ x: 10, y: 10 }).then((svg) => {
  console.log(svg);
});

But I have the following error :

SourceDiagnostic { severity: Error, span: Span(4589265924632), message: "failed to load package (JsValue(ReferenceError: XMLHttpRequest is not defined\nReferenceError: XMLHttpRequest is not defined\n    at FetchPackageRegistry.pullPackageData (/home/millian/node.js/node_modules/@myriaddreamin/typst.ts/dist/cjs/fs/package.cjs:13:25)\n    at FetchPackageRegistry.resolve (/home/millian/node.js/node_modules/@myriaddreamin/typst.ts/dist/cjs/fs/package.cjs:33:27)\n    at ProxyContext.<anonymous> (/home/millian/node.js/node_modules/@myriaddreamin/typst.ts/dist/cjs/options.init.cjs:151:40)\n    at file:///home/millian/node.js/node_modules/@myriaddreamin/typst-ts-web-compiler/pkg/typst_ts_web_compiler.mjs:785:37\n    at handleError (file:///home/millian/node.js/node_modules/@myriaddreamin/typst-ts-web-compiler/pkg/typst_ts_web_compiler.mjs:273:18)\n    at imports.wbg.__wbg_call_01734de55d61e11d (file:///home/millian/node.js/node_modules/@myriaddreamin/typst-ts-web-compiler/pkg/typst_ts_web_compiler.mjs:784:67)\n    at wasm://wasm/031dfe72:wasm-function[3005]:0x6bb41e\n    at wasm://wasm/031dfe72:wasm-function[5378]:0x800d01\n    at wasm://wasm/031dfe72:wasm-function[114]:0xd247d\n    at wasm://wasm/031dfe72:wasm-function[180]:0x1739cb))", trace: [], hints: [] }

The problem is that fetch package use XMLhttpRequest, but that web browser function isn't embedded in Node.js. Maybe I should make a FetchPackageRegistry class. I will try to investigate.

[Myriad-Dreamin]

Hi, @millianlmx sorry about late, I find I was occasionally ignoring the notification email from GitHub... And we are discussing in a closed issue, so I don't find your message till tonight...

I have updated an example for you, it is here:

https://github.com/Myriad-Dreamin/typst.ts/blob/c7b2bb594814d408fae84a63d1b6f7b0d507031c/packages/templates/node.js-compiler-next/src/main.package.ts#L18-L29

Btw, I have made a useful utility library based on typst.ts, which would be more easy to use in most case:

https://github.com/Myriad-Dreamin/typst.ts/blob/c7b2bb594814d408fae84a63d1b6f7b0d507031c/packages/templates/node.js-compiler-next/src/main.snippet.ts#L1-L10

The utility library will be released along with typst.ts v0.4.0-rc9.