Transformer.js Library Returning Error in Browser

Ifeanyi55 commented 10 months ago

Describe the bug A clear and concise description of what the bug is.

I am building a text to text generation app using the Transformer.js library from Hugging Face. However, when I run the app, I get the message "undefined" returned in my div element, and the following error message in my Google Chrome browser console:

transformers@2.7.0:34 2023-11-08 09:38:48.730500 [W:onnxruntime:, graph.cc:3490 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.2/layer.2/DenseReluDense/act/Constant_1_output_0'. It is not used by any node and should be removed from the model.

Running the app in Firefox, I get the source map error message below:

Source map error: Error: request failed with status 404
Resource URL: https://cdn.jsdelivr.net/npm/@xenova/transformers@2.7.0
Source Map URL: transformers.min.js.map

How to reproduce Steps or a minimal working example to reproduce the behavior

I am building with VSCode on a Windows 10 PC, and below is my reproducible code:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Text Generation</title>
    <style>
        .text-class {
            text-align: center;
        }

        #text-output {
            border-width: auto;
            text-align: center;
        }
    </style>
</head>
<body>
    <h1 class="text-class"><strong>Text Generating App</strong></h1>
    <form id="myForm" class="text-class">
        <input type="text" id="inputText">
        <button type="submit" id="submitButton">Submit</button>
    </form>
    <br><br>
    <div id="text-output"></div>
    <script type="module">
        import { pipeline } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers@2.7.0';

        let inputText = document.getElementById("inputText");
        let button = document.getElementById("submitButton");
        let outputText = document.getElementById("text-output");

        button.addEventListener("click", async (e) => {
            // prevent default form submission
            e.preventDefault();
            // specify task
            let generator = await pipeline('text2text-generation');

            let output = await generator(inputText.value, {
                max_new_tokens: 100,
              });

            // show result
            outputText.innerHTML = output.text;

        })
    </script>

</body>
</html>

Expected behavior A clear and concise description of what you expected to happen.

I expect the generated text to be displayed in the div element on the website. Instead, it is "undefined" that is being printed. The javascript code was copied directly from the Hugging Face website.

Logs/screenshots If applicable, add logs/screenshots to help explain your problem.

Environment

Transformers.js version:
Browser (if applicable):
Operating system (if applicable):
Other:

Additional context Add any other context about the problem here.

xenova commented 10 months ago

Hi there. The reason you are seeing undefined is because your usage is incorrect. Please refer to the documentation for example code.

let text = 'I enjoy walking with my cute dog,';
let generator = await pipeline('text-generation', 'Xenova/distilgpt2');
let output = await generator(text);
// [{ generated_text: "I enjoy walking with my cute dog, and I love to play with the other dogs." }]

In your case,

- outputText.innerHTML = output.text
+ outputText.innerHTML = output[0].generated_text

As for the other warnings, you can safely ignore those. A future version of the library will remove those.

Ifeanyi55 commented 10 months ago

Thanks for your response. I corrected the code, but I am getting a new error message stated below:

transformers@2.7.0:97 Unable to load from local path "/models/Xenova/LaMini-Flan-T5-783M/onnx/encoder_model_quantized.onnx": "TypeError: Failed to fetch"
h @ transformers@2.7.0:97
index.html:1 Access to fetch at 'file:///C:/models/Xenova/LaMini-Flan-T5-783M/onnx/decoder_model_merged_quantized.onnx' from origin 'null' has been blocked by CORS policy: Cross origin requests are only supported for protocol schemes: http, data, isolated-app, chrome-extension, chrome, https, chrome-untrusted.
/C:/models/Xenova/LaMini-Flan-T5-783M/onnx/decoder_model_merged_quantized.onnx:1 

       Failed to load resource: net::ERR_FAILED
transformers@2.7.0:97 Unable to load from local path "/models/Xenova/LaMini-Flan-T5-783M/onnx/decoder_model_merged_quantized.onnx": "TypeError: Failed to fetch"
h @ transformers@2.7.0:97

The error also pops up in the Firefox browser as well. What do you recommend as this looks like a protocol compatibility issue. Did you ever encounter an error like this?

xenova commented 10 months ago

When trying to load the models for the first time, the library checks if the model files are stored locally first. If not (which are the 404 errors), the weights will be loaded from the Hugging Face Hub.

You can disable this check as follows:

import { pipeline, env } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers@2.7.0';
env.allowLocalModels = false;

Ifeanyi55 commented 10 months ago

Thanks again man! Made those adjustments to my code but still getting the "undefined" response. Could it have to do with library version? Here is the error in my console in Firefox:


Source map error: Error: request failed with status 404
Resource URL: https://cdn.jsdelivr.net/npm/@xenova/transformers@2.7.0
Source Map URL: transformers.min.js.map

xenova commented 10 months ago

Could you please let me know which model and task you intend to use, as this will change how the output is presented. For example, if you want to use t5-small for text2text-generation, you can write the following code (see demo):

let generator = await pipeline('text2text-generation', 'Xenova/t5-small');
let output = await generator(inputText.value, {
    max_new_tokens: 100,
});

outputText.innerHTML = output[0]; // See note below

For example, now entering "Translate English to French: Hello." will produce "Bonjour."

Also, I would recommend moving the await pipeline(...) call outside of the button action listener, because otherwise, a new pipeline will be created each time (and hence, a lot of memory wasted).

Note: For consistency, the correct usage should be actually output[0].generated_text. This is a mistake of mine, and I will publish a fix for this soon.

Ifeanyi55 commented 10 months ago

I am performing a text2text generation task and I just changed the output[0].generated_text to output[0] which now works. However, the runtime is quite slow. I am using Xenova/LaMini-Flan-T5-783M model.

xenova commented 10 months ago

However, the runtime is quite slow. I am using Xenova/LaMini-Flan-T5-783M model.

Indeed, that is because the model is over 800MB in size. 😅 But once WebGPU support is added, you'll be able to see much greater speedups. For now (and for your demo) I would recommend choosing a smaller model. See here for a list of possible options (but you can always convert your own model if you have one).

Ifeanyi55 commented 10 months ago

Perhaps I have maxed out the browser memory because it is no longer returning any response 😄. I will change to a smaller model. WebGPU support will be a cool upgrade to the library.

xenova commented 10 months ago

Also, I would recommend moving the await pipeline(...) call outside of the button action listener, because otherwise, a new pipeline will be created each time (and hence, a lot of memory wasted).

Just a reminder, as this is probably your problem when running multiple times.

Ifeanyi55 commented 10 months ago

Thanks, will make that adjustment to my code. Thanks for all your help man! Will knock on your door again if I have any future issues building serverless AI applications with transformer.js, if you don't mind 😄

Ifeanyi55 commented 10 months ago

One last question, please. Is this library production-ready?

xenova commented 10 months ago

One last question, please. Is this library production-ready?

That's the goal! 😉 Fortunately, since our API is based on the python transformers library, you shouldn't see massive unexpected changes.

Also, you can always choose to freeze the library at a certain version that works for you (e.g., how Supabase uses v2.5.0 for computing embeddings in edge functions).

Ifeanyi55 commented 10 months ago

Now the very last question on this issue, I promise, and pardon me if it is silly. You suggested running the pipeline outside of the button action listener to reduce memory usage. By that, I am guessing you mean writing the entire async function outside of the button action listener and then calling it inside the listener?

xenova commented 10 months ago

The important part is knowing that await pipeline(...) creates a completely new session, and allocates new memory for the model. So, you should only be running that function once... wherever it may be :)

In your original version, you created a new pipeline on every click (not ideal).

Ifeanyi55 commented 10 months ago

Thanks man!

xenova / transformers.js

Transformer.js Library Returning Error in Browser #383