search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
134.33k stars 26.86k forks source link

Cannot export sdxl encoder to onnx when transformers[torch] >= 4.43.0 (Occurred when translating scaled_dot_product_attention). #33089

Closed puppetpyt closed 1 month ago

puppetpyt commented 2 months ago

System Info

python3.9

Requirement already satisfied: filelock in /usr/local/lib/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (3.13.4)
Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /usr/local/lib/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (0.24.6)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib64/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (1.26.4)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (24.0)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib64/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (6.0.1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib64/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (2023.12.25)
Requirement already satisfied: requests in /usr/local/lib/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (2.31.0)
Requirement already satisfied: tokenizers<0.20,>=0.19 in /home/yuntaopan/.local/lib/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (0.19.1)
Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib64/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (0.4.2)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (4.66.2)
Requirement already satisfied: torch in /usr/local/lib64/python3.9/site-packages (from transformers[torch]==4.43.0) (2.1.2)
Requirement already satisfied: accelerate>=0.21.0 in /usr/local/lib/python3.9/site-packages (from transformers[torch]==4.43.0) (0.29.2)
Requirement already satisfied: psutil in /usr/local/lib64/python3.9/site-packages (from accelerate>=0.21.0->transformers[torch]==4.43.0) (5.9.8)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.9/site-packages (from huggingface-hub<1.0,>=0.23.2->transformers==4.43.0->transformers[torch]==4.43.0) (2024.2.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.9/site-packages (from huggingface-hub<1.0,>=0.23.2->transformers==4.43.0->transformers[torch]==4.43.0) (4.11.0)
Requirement already satisfied: sympy in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (1.12)
Requirement already satisfied: networkx in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (3.2.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (3.1.3)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (12.1.105)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (8.9.2.26)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (12.1.3.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (10.3.2.106)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (11.4.5.107)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (12.1.0.106)
Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /home/yuntaopan/.local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (2.18.1)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (12.1.105)
Requirement already satisfied: triton==2.1.0 in /home/yuntaopan/.local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (2.1.0)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.9/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch->transformers[torch]==4.43.0) (12.4.127)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib64/python3.9/site-packages (from requests->transformers==4.43.0->transformers[torch]==4.43.0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests->transformers==4.43.0->transformers[torch]==4.43.0) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.9/site-packages (from requests->transformers==4.43.0->transformers[torch]==4.43.0) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests->transformers==4.43.0->transformers[torch]==4.43.0) (2024.2.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib64/python3.9/site-packages (from jinja2->torch->transformers[torch]==4.43.0) (2.1.5)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.9/site-packages (from sympy->torch->transformers[torch]==4.43.0) (1.3.0)

Who can help?

No response

Information

Tasks

Reproduction

When I use version 4.42.4 transformers, I can still export.Do I need to make any changes to adapt to version 4.43? export_onnx.py

from dataclasses import dataclass
from diffusers import DiffusionPipeline
from diffusers.utils import BaseOutput
from diffusers.utils.torch_utils import randn_tensor
import torch
import os, sys
from pathlib import Path

from optimum.exporters.tasks import TasksManager
import transformers
import onnx
import shutil
# load both base & refiner
base = DiffusionPipeline.from_pretrained("sdxl/huggingface_model/", torch_dtype=torch.float32, variant = 'float16', use_safetensors=True)
base.to("cpu")

class text_encoder(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = base.text_encoder

    def forward(self, input_ids):
        prompt_embeds = self.encoder(input_ids, output_hidden_states=True)
        return prompt_embeds[0], prompt_embeds.hidden_states[-2]

def export_text_encoder():
    model = text_encoder()
    input = torch.tensor([range(77)], dtype=torch.long)
    output = 'encoder.onnx'

    torch.onnx.export(model, (input),
                        str(output),
                        verbose=False,
                        input_names=['input_ids'],
                        output_names=['pooled_prompt_embed', 'prompt_embed'],
                        #dynamic_axes={"input": { 1: "seq_len" } },
                        do_constant_folding=True,
                        opset_version=15)
if __name__ == '__main__':

    export_text_encoder()

Traceback (most recent call last): File "/home/export_onnx.py", line 59, in export_text_encoder() File "/home/export_onnx.py", line 49, in export_text_encoder torch.onnx.export(model, (input), File "/usr/local/lib64/python3.9/site-packages/torch/onnx/utils.py", line 516, in export _export( File "/usr/local/lib64/python3.9/site-packages/torch/onnx/utils.py", line 1596, in _export graph, params_dict, torch_out = _model_to_graph( File "/usr/local/lib64/python3.9/site-packages/torch/onnx/utils.py", line 1139, in _model_to_graph graph = _optimize_graph( File "/usr/local/lib64/python3.9/site-packages/torch/onnx/utils.py", line 677, in _optimize_graph graph = _C._jit_pass_onnx(graph, operator_export_type) File "/usr/local/lib64/python3.9/site-packages/torch/onnx/utils.py", line 1940, in _run_symbolic_function return symbolic_fn(graph_context, *inputs, attrs) File "/usr/local/lib64/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 306, in wrapper return fn(g, *args, *kwargs) File "/usr/local/lib64/python3.9/site-packages/torch/onnx/symbolic_opset14.py", line 176, in scaled_dot_product_attention query_scaled = g.op("Mul", query, g.op("Sqrt", scale)) File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 87, in op return _add_op(self, opname, raw_args, outputs=outputs, kwargs) File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 238, in _add_op inputs = [_const_if_tensor(graph_context, arg) for arg in args] File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 238, in inputs = [_const_if_tensor(graph_context, arg) for arg in args] File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 269, in _const_if_tensor return _add_op(graph_context, "onnx::Constant", value_z=arg) File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 246, in _add_op node = _create_node( File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 305, in _create_node _add_attribute(node, key, value, aten=aten) File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 356, in _addattribute return getattr(node, f"{kind}")(name, value) TypeError: z_(): incompatible function arguments. The following argument types are supported:

  1. (self: torch._C.Node, arg0: str, arg1: torch.Tensor) -> torch._C.Node

Invoked with: %354 : Tensor = onnx::Constant(), scope: main.text_encoder::/transformers.models.clip.modeling_clip.CLIPTextModel::encoder/transformers.models.clip.modeling_clip.CLIPTextTransformer::text_model/transformers.models.clip.modeling_clip.CLIPEncoder::encoder/transformers.models.clip.modeling_clip.CLIPEncoderLayer::layers.0/transformers.models.clip.modeling_clip.CLIPSdpaAttention::self_attn , 'value', 0.125 (Occurred when translating scaled_dot_product_attention).

Expected behavior

Successfully exported onnx

ArthurZucker commented 2 months ago

cc @IlyasMoutawwakil

IlyasMoutawwakil commented 2 months ago

the change I had to do in optimum https://github.com/huggingface/optimum/pull/1971 is to use the eager attention since you can't foward attn_implementation="eager" in a diffusion pipeline to the clip model https://github.com/huggingface/diffusers/issues/8957. I still haven't investigated why sdpa can't be exported in this case.

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.