Running Translation models in v3 throws an error

0xWOLAND commented 7 months ago

System Info

@xenova/transformers": "github:xenova/transformers.js#v3

Using PopOs 22.04 in a Node.js environment

Environment/Platform

[ ] Website/web-app
[ ] Browser extension
[X] Server-side (e.g., Node.js, Deno, Bun)
[ ] Desktop app (e.g., Electron)
[ ] Other (e.g., VSCode extension)

Description

Running

    // Create translation pipeline
    let translator = await pipeline(
      "translation",
      "Xenova/mbart-large-50-many-to-many-mmt"
    );

    console.log("translator -- ", translator);

    // Translate text from Hindi to French
    let output = await translator(
      "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है",
      {
        src_lang: "hi_IN", // Hindi
        tgt_lang: "fr_XX", // French
      }
    );

    // [{ translation_text: 'Le chef des Nations affirme qu 'il n 'y a military solution in Syria.' }]
    console.log(output);

In the main branch of transformers.js works, but it throws


  TypeError {
    message: 'this.ort_tensor.dispose is not a function',
  }

in the v3 branch (#545)

Reproduction

Run

    // Create translation pipeline
    let translator = await pipeline(
      "translation",
      "Xenova/mbart-large-50-many-to-many-mmt"
    );

    console.log("translator -- ", translator);

    // Translate text from Hindi to French
    let output = await translator(
      "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है",
      {
        src_lang: "hi_IN", // Hindi
        tgt_lang: "fr_XX", // French
      }
    );

    // [{ translation_text: 'Le chef des Nations affirme qu 'il n 'y a military solution in Syria.' }]
    console.log(output);

using the v3 branch

0xWOLAND commented 7 months ago

This issue also exists for 'Xenova/mbart-large-50-many-to-many-mmt' and 'distil-whisper/distil-large-v3'

xenova commented 7 months ago

Thanks for the report. We haven't yet got encoder-decoder models working with v3, so this error is expected (at least for now). I will close this issue once we do get it working though :)

0xWOLAND commented 7 months ago

@xenova thank you! Do you happen to have a rough timeline for when encoder-decoder models will be offered with v3?

calumk commented 7 months ago

Ah, Ok this seems to maybe cause my issue too?

I was trying with nnlb-200-distilled-600M and m2m100_418M

When i run await pipeline it just hangs forever.

const translator = await pipeline('translation', 'Xenova/nllb-200-distilled-600M');

weirdly with an old version of nllb-200-distilled-600M it doesnt hang....

xenova commented 7 months ago

@0xWOLAND We are actively working on this, but for the most part, it will depend on the release of onnxruntime v1.17.3, which fixes several issues with these models.

@calumk Are you using Transformers.js v3? Do note that this is a very large model, and the hanging is most likely due to the downloading of the weights. Please check your network tab to verify this guess.

calumk commented 7 months ago

@xenova I am using Bun, with transformers.js v3 and locally downloaded models.

I had no issues doing it with node, transformers.js v2, I dont think its a downloading issue, but i dont think there is a way to get any information out of "pipeline" to see why?

Either way, if the translation models are not yet supported in v3, then it doesnt matter yet, as i would get any further anyway

For The sake of clairty / logging, my problem is below, but dont feel like you need to waste time on it.

If i load "nllb-200-distilled-600M_old" (maybe downloaded some time around november last year?)

I run const translator = await pipeline('translation', 'Xenova/nllb-200-distilled-600M_old');

I get some debug in the terminal (But i always had these errors and it is just messy debug, but not a problem)

Removing initializer '/model/decoder/layers.1/final_layer_norm/Constant_1_output_0'. It is not used by any node and should be removed from the model.

The model then loads
The translator command doesnt work and hangs forever

if i load "nllb-200-distilled-600M" (downloaded today)

I run const translator = await pipeline('translation', 'Xenova/nllb-200-distilled-600M');
It hangs here, and never progresses

The Models dont seem to have version numbers as far as i can tell - so the best i can provide to understand is the screenshot :

xenova commented 7 months ago

As seen here, the model you are loading (quantized or fp32) haven't been updated in 7 months, so it shouldn't be an issue with that.

You could maybe check if the .onnx files (in the onnx subfolder) are exactly the same.

Either way, if the translation models are not yet supported in v3, then it doesnt matter yet, as i would get any further anyway

Right, and at the moment, we actually throw an error when trying to call the pipeline: https://github.com/xenova/transformers.js/pull/545/files#diff-2f6b66f61363f7b45e1b165f81d3ce15b3768da43e40410085aee8bd8666a629R681

calumk commented 6 months ago

@xenova

It seems onnxruntime v1.17.3 has been released a few weeks ago

Does that bring us closer to encoder-decoder models in v3?

flatsiedatsie commented 2 months ago

I just tried it, but it still doesn't seem to run in V3 (alpha 9, Brave browser)

It does run under V2. It's slow, but I gotta say, it's good! Translation quality seems better than with OPUS single-language models. Though those are lightning fast with WebGPU.

huggingface / transformers.js