[Feature request] Deno Support

[omar2205]

Name of the feature Support running in the Deno runtime.

Additional context I tried all the tricks, esm.sh, unpkg, esm.run, and npm:@xenova/transformers, but nothing worked.

Using unpkg import I get

This browser lacks typed array (Uint8Array) support which is required 
by `buffer` v5.x. Use `buffer` v4.x if you require old browser support.

Using esm.sh

Uncaught SyntaxError: The requested module '/v114/pngjs@6.0.0/deno/pngjs.mjs' 
does not provide an export named 'PNG'

[xenova]

This seems to be an problem with onnxruntime itself: https://github.com/microsoft/onnxruntime/issues/10913

The person who opened the issue did seem to find a workaround, so, maybe that will help too?

@josephrocca is quite active in the community, and since he opened up that issue, he might have some more insight.

[omar2205]

So @nestarz created a Deno onnx_runtime here https://github.com/nestarz/onnx_runtime

[xenova]

So @nestarz created a Deno onnx_runtime here https://github.com/nestarz/onnx_runtime

That's good to know - thanks! Have you tried overriding the dependency in your project?

[omar2205]

Not sure how. For example, this is my script file

  import * as t from 'npm:@xenova/transformers'

  const pipeline = t.default.pipeline

  let pipe = await pipeline('sentiment-analysis')

  let out = await pipe('I love transformers!')

This gets a new error, error: Uncaught Error: Not implemented: Worker from onnxruntime-web/1.14.0/dist/ort-web.node.js

[xenova]

Maybe this might help: https://www.reddit.com/r/Deno/comments/x9q9vp/how_do_you_patch_a_module_like_patchpackage_in/ ? I haven't used Deno before, so, I probably won't be able to provide much more help (other than googling or "chatgpt-ing" 🤣 )

[xenova]

Have you been able to get it working in Deno yet? If not, feel free to try with the latest version 2.0.0-alpha.0. This may fix it.

[omar2205]

Hey, I just tried it, using npm: and esm.sh and both got an error. esm.sh:

Uncaught Error: Dynamic require of "../bin/napi-v3/linux/x64/onnxruntime_binding.node" is not supported

npm:

Uncaught Error:                                                                 
Something went wrong installing the "sharp" module                              

Cannot find module '../build/Release/sharp-linux-x64.node'
Require stack:

[takoyaro]

it looks like sharp doesn't support deno at the moment

[xenova]

@omar2205 Have you tried the steps outlined here: https://github.com/lovell/sharp/issues/2583#issuecomment-1310628079 ?

This is an apparent workaround until https://github.com/denoland/deno/issues/16164 is made available.

[omar2205]

Thanks for the ping @xenova, I followed that and it worked, however, I'm met with Invalid URL: error.

import { pipeline } from 'npm:@xenova/transformers';
/* warning
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
napi_add_finalizer is not yet supported.
*/

let pipe = await pipeline('sentiment-analysis');

/* Error
No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
Uncaught TypeError: Invalid URL: 'Xenova/distilbert-base-uncased-finetuned-sst-2-english/tokenizer.json'
*/

[xenova]

Okay great! That error is probably deno specific, because the check to see whether something is a valid URL is surrounded by a try-catch block (and works in other environments).

https://github.com/xenova/transformers.js/blob/0b403ce8e4cf4ab447f8f3b208920aa4e77dc499/src/utils/hub.js#L171-L179

While I look into it, you can get around it temporarily by disabling local models:

import { pipeline, env } from 'npm:@xenova/transformers';

env.allowLocalModels=false;

let pipe = await pipeline('sentiment-analysis');

[omar2205]

I tried to do that before, but it fails:

No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
Uncaught TypeError: Invalid URL: 'Xenova/distilbert-base-uncased-finetuned-sst-2-english/tokenizer.json'

[xenova]

Hmm, it must be failing elsewhere then. Are you able to send the full stack trace?

[omar2205]

Here you go

No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
Uncaught TypeError: Invalid URL: 'Xenova/distilbert-base-uncased-finetuned-sst-2-english/tokenizer.json'
    at getSerialization (ext:deno_url/00_url.js:88:11)
    at new URL (ext:deno_url/00_url.js:383:27)
    at new Request (ext:deno_fetch/23_request.js:297:25)
    at Cache.[[[matchAll]]] (ext:deno_cache/01_cache.js:230:11)
    at Cache.match (ext:deno_cache/01_cache.js:173:36)
    at getModelFile (file:///home/ubuntu/.cache/deno/npm/registry.npmjs.org/@xenova/transformers/2.0.0-alpha.2/src/utils/hub.js:347:32)
    at eventLoopTick (ext:core/01_core.js:166:11)
    at async getModelJSON (file:///home/ubuntu/.cache/deno/npm/registry.npmjs.org/@xenova/transformers/2.0.0-alpha.2/src/utils/hub.js:456:18)
    at async Promise.all (index 0)
    at async loadTokenizer (file:///home/ubuntu/.cache/deno/npm/registry.npmjs.org/@xenova/transformers/2.0.0-alpha.2/src/tokenizers.js:49:16)

[xenova]

Thanks! It looks like deno has it's own caching API, which only accepts HTTP URLs. I'll see what I can do.

[xenova]

In the meantime, can you try disabling the cache system? You can do this by setting: env.useBrowserCache=false;

[omar2205]

That fixed it. But we got another error 😅

> let out = await pipe('I love transformers!');

An error occurred during model execution: "TypeError: Tensor.data must be a typed array (9) for int64 tensors, but got typed array (-1).".
Inputs given to model: {
  input_ids: Tensor {
    dims: [ 1, 6 ],
    type: "int64",
    data: BigInt64Array(6) [ 101n, 1045n, 2293n, 19081n, 999n, 102n ],
    size: 6
  },
  attention_mask: Tensor {
    dims: [ 1, 6 ],
    type: "int64",
    data: BigInt64Array(6) [ 1n, 1n, 1n, 1n, 1n, 1n ],
    size: 6
  }
}
Uncaught TypeError: Tensor.data must be a typed array (9) for int64 tensors, but got typed array (-1).
    at file:///home/ubuntu/.cache/deno/npm/registry.npmjs.org/onnxruntime-node/1.14.0/dist/backend.js:45:108
    at Array.processTicksAndRejections (ext:deno_node/_next_tick.ts:23:21)
    at eventLoopTick (ext:core/01_core.js:178:29)

[xenova]

BigInt64Array is a typed array, so, I'm not too sure what the error message means... 👀 This is most likely a limitation of onnxruntime-web/onnxruntime-node :/

[omar2205]

Yeah looks like it. Is it possible to use this https://github.com/nestarz/onnx_runtime?

[xenova]

If you'd like, you can fork this repo and replace the onnxruntime import with that?

[omar2205]

After second thought, I don't know how that would even work. It's written for Deno.

[xenova]

They seem to provide an import syntax in their README:

import * as ort from "https://deno.land/x/onnx_runtime/mod.ts";

You can edit the ONNX imports in https://github.com/xenova/transformers.js/blob/main/src/backends/onnx.js

[josephrocca]

Maybe I'm missing something, but: Wouldn't it make more sense to use the browser version of onnxruntime for Deno? Deno is aimed at being strongly web-compatible, so running transformers.js/onnxruntime with node.js emulation (i.e. importing with npm: rather than e.g. jsdelivr) seems like a very round-about way that would result in more problems? Again, might be missing something as I have only skimmed this issue.

[omar2205]

Trying using jsdelivr gives Uncaught DOMException error

[xenova]

I set up a codespace and got it working:

import { pipeline, env } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers';
// import {pipeline, env} from '@xenova/transformers';

env.useBrowserCache=false;
env.allowLocalModels=false;

let pipe = await pipeline('text-classification')

let output = await pipe('I love Transformers.js')

console.log(output)

console output:

$ cat main.js | deno run --allow-all --unstable -
No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
[ { label: "POSITIVE", score: 0.99961256980896 } ]

---

The only drawback with this approach is that you might experience a performance hit since it's running with the WASM backend (and not native CPU). I'll look into why the tensor types are giving errors when running the native version.

[omar2205]

I got Uncaught DOMException when I opened this issue and tried it yesterday But it looks like it worked fine with you, and in this replit https://replit.com/@omar22051/IncompleteMenacingPattern#index.ts

When I tried it in a deno deploy, I got hit with the memory limit.

[xenova]

I got Uncaught DOMException when I opened this issue and tried it yesterday But it looks like it worked fine with you, and in this replit https://replit.com/@omar22051/IncompleteMenacingPattern#index.ts

I don't quite know where Transformers.js would be throwing a DOMException 👀 Is it maybe an issue with your environment (since it works in that replit)?

When I tried it in a deno deploy, I got hit with the memory limit.

Do you know what the memory limit is for deno deploy? Some sources suggest either 256MB or 512MB. The model you're testing with isn't that large.

[omar2205]

I believe it's 512 MB. I managed to get a QA bot going, deploy to Deno Deploy here, which was the demo I was going for.

I don't know if you want to close this issue or wait for onnx to support Deno.

[xenova]

Cool! I'll try look into the previous issue and see if there's a workaround since it might take a while for them to support Deno properly. I think it will also be good to create a Deno tutorial/example project, so, I'll close the issue after I make that.

[birkskyum]

Related to

• https://github.com/denoland/deno/issues/20071

[kivlor]

@xenova I gave your codespace example a try locally with Deno 1.37.0 and I get the following:

No model specified. Using default model: "Xenova/distilbert-base-uncased-finetuned-sst-2-english".
error: Uncaught (in promise) NotSupported: Classic workers are not supported.
    at createWorker (ext:runtime/11_workers.js:41:14)
    at new Worker (ext:runtime/11_workers.js:110:16)
    at Object.yc (https://cdn.jsdelivr.net/npm/@xenova/transformers:34:7988)
    at Object.Cc (https://cdn.jsdelivr.net/npm/@xenova/transformers:34:8046)
    at de (https://cdn.jsdelivr.net/npm/@xenova/transformers:34:5797)
    at Ae (https://cdn.jsdelivr.net/npm/@xenova/transformers:34:9771)
    at <anonymous> (https://cdn.jsdelivr.net/npm/@xenova/transformers@2.6.1/dist/ort-wasm-simd-threaded.wasm:1:8324360)
    at <anonymous> (https://cdn.jsdelivr.net/npm/@xenova/transformers@2.6.1/dist/ort-wasm-simd-threaded.wasm:1:999116)
    at <anonymous> (https://cdn.jsdelivr.net/npm/@xenova/transformers@2.6.1/dist/ort-wasm-simd-threaded.wasm:1:1913327)
    at <anonymous> (https://cdn.jsdelivr.net/npm/@xenova/transformers@2.6.1/dist/ort-wasm-simd-threaded.wasm:1:4813118)

Any quick ideas? I'll happily dig in the mean time :)

[xenova]

You could try just extract the worker.js logic out into the main script. If the error is happening within onnxruntime, it might be fixable with env.backends.onnx.wasm.numThreads=1 (see here/docs)

[kivlor]

Thanks @xenova, I've got it running now with the following code:

import { pipeline, env } from "https://cdn.jsdelivr.net/npm/@xenova/transformers";

env.backends.onnx.wasm.numThreads = 1;

const classifier = await pipeline("text-classification", "Xenova/distilbert-base-uncased-finetuned-sst-2-english");

const input = ["I love Transformers.js", "I hate Transformers.js", "I not sure about Transformers.js"];
const result = await classifier(input);

console.log(result);

then running in 1.37.1 with:

$ deno run -A main.ts

[devoutdrawai]

I am running into TypeError: c is not a function at Function.fromBlob (https://cdn.jsdelivr.net/npm/@xenova/transformers:100:1417) when trying to use RawImage.fromBlob, is there a work around for this?

[xenova]

@devoutdrawai can you provide the code snippet you are using? Also, if you import the unminified version of the script (https://cdn.jsdelivr.net/npm/@xenova/transformers@2.7.0/dist/transformers.js), you'll see a more helpful error message.

Regardless, I believe you'll be running into issues since deno doesn't yet support offscreen canvas (https://github.com/denoland/deno/issues/19533)

[devoutdrawai]

@xenova Yeah I got Error: OffscreenCanvas not supported by this browser. after trying to create a new RawImage class rather than a blob. Is there a workaround way to pass in a image for inferencing that I just don't know about?

[xenova]

Update from the sharp.js team about installation with Deno: https://github.com/lovell/sharp/issues/3750#issuecomment-1805386175 🥳

A new sharp v0.33.0-alpha.11 pre-release is now available, which has been tested as working out of the box with Deno and has (drum roll please) a WebAssembly version - please see #3750 (comment)

Will be a good idea to revisit proper Deno support (instead of using the web/WASM version).

[birkskyum]

@xenova , I think that's a good idea too to avoid paying the sometimes steep wasm price, unless strictly necessary

Sharp - WASM PR: As you can see, Wasm performs at around native speed in some benchmarks, but is ~2x slower where SIMD is required - which is expected, given that libvips heavily relies on runtime SIMD code generation, which is not yet supported for Wasm.

[kyeshmz]

It seems v0.33 is now publically avaliable!

[birkskyum]

Have this been working since transformers.js 2.9 where sharp was bumped to v0.33?

[DuncanLHS]

transformers.js v 2.13.1

Not sure if this should be a new issue but is also Deno related. I've been using remote models without issue in a Supabase edge function (Deno environment), now experimenting with using the same model locally and hitting errors related to the local filesystem.

I'm getting Invalid URL error

[Error] Unable to load from local path "../_models/Supabase/gte-small/tokenizer.json": "TypeError: Invalid URL: '../_models/Supabase/gte-small/tokenizer.json'"

I've had a pick through the source and don't think this should be happening if the 'fs' package is available, which AFAIK it should be in deno.

So I tried adding env.FS = true and got a different error, again appears related to fs (or lack thereof)

[Error] Unable to load from local path "../_models/Supabase/gte-small/tokenizer.json": "TypeError: r.existsSync is not a function"

Folder structure is: ,,/functions/_models/Supabase/gte-small - model files are here ../functions/embed/embed.ts - pipeline is here

embed.ts

env.useBrowserCache = false
env.allowLocalModels = true
env.allowRemoteModels = false
env.localModelPath = '../_models'
env.useFS = true

const generateEmbedding = await pipeline(
  'feature-extraction',
  'Supabase/gte-small'
)

[xenova]

@DuncanLHS Could you try test if fs is available (outside of transformers.js)? e.g., open a local text file.

[DuncanLHS]

@xenova Yes, but before I do I've just spotted that any node packages available in deno need to use the 'node:' specifier. https://docs.deno.com/deploy/api/runtime-node

[xenova]

Also note that filesystem access is disabled when loading the library from a CDN, so you would need to wait until the library functions correctly with Deno.

[DuncanLHS]

OK, I'll move the functionality into node for now.

[cotyhamilton]

The alpha is working in deno https://huggingface.co/posts/Xenova/681836693682285

working example

DENO_FUTURE=1 deno add npm:@huggingface/transformers@3.0.0-alpha.14

// main.ts
import { pipeline } from "@huggingface/transformers";

async function main() {
  const generateEmbeddings = await pipeline("feature-extraction");
  const embeddings = await generateEmbeddings("Hello, World!");
  console.log(embeddings);
}

if (import.meta.main) {
  main();
}

DENO_FUTURE=1 deno run -RWN --allow-ffi --deny-env main.ts

No model specified. Using default model: "Xenova/all-MiniLM-L6-v2".
dtype not specified for "model". Using the default dtype (fp32) for this device (cpu).
Tensor {
  ort_tensor: Tensor {
    cpuData: Float32Array(2304) [
        0.09704522788524628,   0.04466759413480759,   0.03342921659350395,
         0.2942180037498474,  -0.09638233482837677,  -0.39724844694137573,
       0.029586339369416237,  -0.09695737808942795,    -0.233986034989357,
       -0.11687380075454712,  0.031189898028969765,   0.28859788179397583,
        0.12789778411388397,  0.007472040131688118,   0.09201933443546295,
        -0.2740094065666199,   -0.0917656198143959,   0.09536977857351303,
       0.011544985696673393,  -0.07689643651247025,  -0.08625669777393341,
        -0.0317402184009552,   0.19354557991027832,  0.056520186364650726,
       -0.18216514587402344,    0.1443641483783722,  -0.06378541141748428,
        0.22648848593235016,  -0.09405055642127991,   -0.7146937251091003,
       -0.02438972145318985,   -0.1423027515411377,    0.3438223600387573,
       -0.24876512587070465,  -0.09658709168434143,    0.3035530149936676,
        0.04076158255338669,   -0.2887788712978363,   -0.1621636301279068,
        0.07270413637161255,    0.1354200392961502,  -0.18576160073280334,
       -0.11635178327560425,   0.14100773632526398,   0.03162102401256561,
       -0.01900934800505638,  -0.21902813017368317,   0.23496025800704956,
       0.041369009763002396,    0.1376398801803589,   -0.3159908354282379,
       -0.05101257562637329,   0.08553100377321243,   -0.1915268898010254,
         0.1287413239479065,   0.12561288475990295, -0.013958334922790527,
        0.09618371725082397, -0.046709202229976654,  -0.19115103781223297,
       -0.04283100366592407,  -0.09430375695228577,  -0.23719236254692078,
      -0.027194540947675705,   0.06842745095491409,   -0.3157595992088318,
        0.08512159436941147,   0.10566434264183044, -0.053132228553295135,
        -0.0351015068590641,  -0.15570200979709625,    0.0818176120519638,
         0.1027064248919487,   0.04647187143564224,   -0.1836329847574234,
       -0.11254461109638214,   0.06946499645709991,   0.02888619899749756,
         0.2119930386543274,    0.0845736488699913, -0.038595687597990036,
        0.02478068694472313,   0.09258890151977539, -0.021482625976204872,
       -0.05618399381637573,   0.15647916495800018,   0.00866264570504427,
        0.03660000488162041,  -0.12824304401874542,  -0.09475530683994293,
       -0.47616755962371826,  -0.18034975230693817,  0.041318655014038086,
           0.09013681858778,   -0.6151973009109497,  0.008081695064902306,
        0.11617103964090347,  -0.17217235267162323,   -0.3506837487220764,
           5.51707649230957,
      ... 2204 more items
    ],
    dataLocation: "cpu",
    type: "float32",
    dims: [ 1, 6, 384 ],
    size: 2304
  }
}

Unfortunately only able to use cpu device though