Error when use .onnx on web

tidus2102 commented 2 months ago

Hi, I tried to test .onnx on web (CPU) using onnx-community/BiRefNet_T. Here is index.html:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <title>Background Removal</title>
</head>
<body>
<script type="module">
  import { AutoModel, AutoProcessor, RawImage } from 'https://cdn.jsdelivr.net/npm/@huggingface/transformers';

  // Load model and processor
  const model_id = 'onnx-community/BiRefNet_T';
  const model = await AutoModel.from_pretrained(model_id, { dtype: 'fp32' });
  const processor = await AutoProcessor.from_pretrained(model_id);

  // Load image from URL
  const url = 'https://images.pexels.com/photos/5965592/pexels-photo-5965592.jpeg?auto=compress&cs=tinysrgb&w=1024';
  const image = await RawImage.fromURL(url);

  // Pre-process image
  const { pixel_values } = await processor(image);

  // Predict alpha matte
  const { output_image } = await model({ input_image: pixel_values });
  console.log(output_image);
</script>
</body>
</html>

Check Chrome dev tools > Application tab > Cache storage, the .onnx file is loaded & cached without problem: Screenshot 2024-08-28 at 20 07 41

But I got error at predict code line:

const { output_image } = await model({ input_image: pixel_values })

hiLBrmNL_G8aENlwRHGGQ

I try the other input images, got the same issue. Please help to check it. Thank you.

ZhengPeng7 commented 2 months ago

Thanks for letting me know about this problem. I'm not familiar with the web app, but I will look into it tomorrow and reply to you then.

ZhengPeng7 commented 2 months ago

Have you tried following the example codes in my HF space demo? You can click the Use via API at the bottom of the page and follow the instructions there (with an uploaded image or a URL to the online image):

截屏2024-08-29 14 07 55

tidus2102 commented 2 months ago

Thanks for your reply. I already tried your demo spaces and it works. I only ask about issue when using .onnx file on js (using transformers.js). I haven't tried BiRefNet .onnx on python but saw you and someone have tried it here.

About .onnx on js, I tested the briaai/RMBG-1.4, it works without any issues. So I'm not sure if BiRefNet .onnx file got some issues. I haven't tried create .onnx file again from pytorch file on my local (I'm using Macbook M1).

ZhengPeng7 commented 2 months ago

Please use the official ONNX files (you can download them from the GitHub release or GDrive folder given in README). I'm not saying their ONNX files are wrong, but I don't know whether there is any problem.

ZhengPeng7 commented 2 months ago

Hi, Hung, I tried to learn about this problem. I also saw that you raised a discussion in the ONNX-community's HF repo. I really appreciate it, but I'm really not good at (even a noob) web deployment (things like transformers.js). Very sorry for that. If I'm not so busy with other work in recent days, I'll try to learn and solve it step by step.

Could you send me your demo projects, which I could build from zero, so I can test the error you encounter? Many thanks! If I solve it by myself, you'll get notified here. Finally, let me add a need-help badge to this issue to see if any other guy is good at it and do some help.

tidus2102 commented 2 months ago

hi, sorry for the late reply, I tried to find out more about this issue after got your reply.

First, the .onnx file on onnx-community/BiRefNet_T is the same with your .onnx on github releases.

Second, here is my demo code: https://codesandbox.io/p/sandbox/birefnet-onnx-5c7vv8

NOTE:

running BiRefNet_T model takes longer time than RMBG-1.4 (on my Macbook M1, it takes about 20 seconds then raise error)
when run (multiple times) the Chrome tab may consume very high RAM, it will crash if your machine doesn't have enough RAM, if it crashed just open the tab again

Demo run: Screenshot 2024-09-01 at 14 08 48

Check Chrome dev tools > Console tab to see the model config (both models) after model loaded: onnx-community/BiRefNet_T .onnx config:

{
    "model_type": "swin",
    "is_encoder_decoder": false,
    "normalized_config": {
        "model_type": "swin",
        "is_encoder_decoder": false
    }
}

briaai/RMBG-1.4 .onnx config:

{
    "model_type": "SegformerForSemanticSegmentation",
    "is_encoder_decoder": false,
    "_name_or_path": "briaai/RMBG-1.4",
    "architectures": [
        "BriaRMBG"
    ],
    "auto_map": {
        "AutoConfig": "MyConfig.RMBGConfig",
        "AutoModelForImageSegmentation": "briarmbg.BriaRMBG"
    },
    "custom_pipelines": {
        "image-segmentation": {
            "impl": "MyPipe.RMBGPipe",
            "pt": [
                "AutoModelForImageSegmentation"
            ],
            "tf": [],
            "type": "image"
        }
    },
    "in_ch": 3,
    "out_ch": 1,
    "torch_dtype": "float32",
    "transformers_version": "4.38.0.dev0",
    "normalized_config": {
        "model_type": "SegformerForSemanticSegmentation",
        "is_encoder_decoder": false
    }
}

Error when run predict with BiRefNet .onnx Screenshot 2024-09-01 at 14 04 03

Check Chrome dev tools > Application tab > Cache storage > transformers-cache for .onnx model files load from huggingface: Screenshot 2024-09-01 at 13 54 46

Third:

I tried to create my own .onnx file (general lite weight) on google colab, I modified some of your demo code so my output .onnx different with your .onnx but still got the same error when use with js.
I tried to use your .onnx & mine with python => both works, no problem.

Summary: the BiRefNet .onnx only got error with js, I see significant differences when log 2 models config. Hope this helps. Thank you.

xenova commented 1 month ago

Hi there 👋 The model (with the example code) runs fine in Node.js, so if I had to guess, it is an out of memory issue due to browser resource constraints. I think there are two fixes we can consider:

Export at a lower input shape (instead of 1024x1024)
Export with dynamic shapes