search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.61k stars 2.92k forks source link

Failed to create CUDAExecutionProvider #13139

Open Matthieu-Tinycoaching opened 2 years ago

Matthieu-Tinycoaching commented 2 years ago

Describe the issue

I compared inference on GPU of a native torch Helsinki-NLP/opus-mt-fr-en model with respect to the optimized onnx model thanks to Optimum library.

When load testing the model on my local computer, I was surprised by two things:

  1. The performance on GPU of the optimized ONNX model is worse than the native torch:

GPU_optimized_onnxruntime GPU_torch

  1. When running this fastAPI service into a docker image I got the following warning:

2022-09-28 08:20:21.214094612 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:566 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.

Does this mean the CUDAExecutionProvider is not working even if I set it in ORTModelForSeq2SeqLM object?

What could be caused that? I saw in https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html that CUDA 11.6 is not mentionned, could it be this?

To reproduce

I have defined a fastAPI microservice based on two classes below for GPU both torch and optimized ONNX, repsectively:

class Seq2SeqModel:
    tokenizer: Optional[MarianTokenizer]
    model: Optional[MarianMTModel]

    def load_model(self):
        """Loads the model"""
        # model_id="Helsinki-NLP/opus-mt-fr-en"
        model_path = Path("./app/artifacts/HF")
        tokenizer = AutoTokenizer.from_pretrained(model_path)
        model = AutoModelForSeq2SeqLM.from_pretrained(model_path).to("cuda")
        self.tokenizer = tokenizer
        self.model = model

    async def predict(self, input: PredictionInput) -> PredictionOutput:
        """Runs a prediction"""
        if not self.tokenizer or not self.model:
            raise RuntimeError("Model is not loaded")
        tokens = self.tokenizer(input.text, return_tensors="pt").to("cuda")
        translated = self.model.generate(**tokens, num_beams=beam_size)
        return PredictionOutput(translated_text=self.tokenizer.decode(translated[0], skip_special_tokens=True))

class OnnxOptimizedSeq2SeqModel:
    tokenizer: Optional[MarianTokenizer]
    model: Optional[ORTModelForSeq2SeqLM]

    def load_model(self):
        """Loads the model"""
        # model_id="Helsinki-NLP/opus-mt-fr-en"
        onnx_path = Path("./app/artifacts/OL_1")
        tokenizer = AutoTokenizer.from_pretrained(onnx_path)
        optimized_model = ORTModelForSeq2SeqLM.from_pretrained(
            onnx_path,
            encoder_file_name="encoder_model_optimized.onnx",
            decoder_file_name="decoder_model_optimized.onnx",
            decoder_file_with_past_name="decoder_with_past_model_optimized.onnx",
            provider="CUDAExecutionProvider"
        )
        self.tokenizer = tokenizer
        self.model = optimized_model

app = FastAPI()
seq2seq_model = Seq2SeqModel()
onnx_optimized_seq2seq_model = OnnxOptimizedSeq2SeqModel()
beam_size = 3

@app.on_event("startup")
async def startup():
    seq2seq_model.load_model()
    onnx_optimized_seq2seq_model.load_model()

@app.post("/prediction")
async def prediction(
    output: PredictionOutput = Depends(seq2seq_model.predict),
) -> PredictionOutput:
    return output

@app.post("/prediction_onnx_optimized")
async def prediction(
    output: PredictionOutput = Depends(onnx_optimized_seq2seq_model.predict),
) -> PredictionOutput:
    return output

Urgency

urgent

Platform

Linux

OS Version

Ubuntu 18.0.4 LTS

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

onnxruntime-gpu: 1.12.1

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 11.6

fxmarty commented 2 years ago

HI @Matthieu-Tinycoaching , don't hesitate to open an issue in Optimum repo if you think it's related as well. Could you tell me if you use onnxruntime-gpu or onnxruntime? Did you install CUDA toolkit and cuDNN following Nvidia guides?

The error

2022-09-28 08:20:21.214094612 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:566 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.

does not mean that you can not use ORTModelForSeq2SeqLM with you GPU. It simply means that there is something wrong in your install of CUDA / onnxruntime-gpu. Passing provider="CUDAExecutionProvider" is supported in Optimum.

I am currently looking into the runtime issues, as it was already reported, stay tuned.

Also, we plan to include more documentation on how to use Optimum-ONNX Runtime on GPU with CUDAExecutionProvider and TensorrtExecutionProvider soon.

fxmarty commented 2 years ago

In my case,

import onnxruntime

from optimum.onnxruntime import ORTModelForSeq2SeqLM

from transformers import MarianTokenizer

tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-fr-en")

options = onnxruntime.SessionOptions()
options.log_severity_level = 0  # verbose, to see which execution provider is used

ort_model = ORTModelForSeq2SeqLM.from_pretrained(
    "Helsinki-NLP/opus-mt-fr-en",
    from_transformers=True,
    provider="CUDAExecutionProvider",
    session_options=options,
)

print(ort_model.providers)

prints the following below. It means that some operations use indeed the CUDAExecutionProvider, and that the execution provider is well loaded. You can use options.log_severity_level = 0 to troubleshoot and check which operators use the GPU and which do not.

2022-09-28 16:05:51.514299: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
/home/fxmarty/hf_internship/transformers/src/transformers/models/marian/modeling_marian.py:234: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/home/fxmarty/hf_internship/transformers/src/transformers/models/marian/modeling_marian.py:241: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attention_mask.size() != (bsz, 1, tgt_len, src_len):
/home/fxmarty/hf_internship/transformers/src/transformers/models/marian/modeling_marian.py:273: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
/home/fxmarty/hf_internship/transformers/src/transformers/models/marian/modeling_marian.py:856: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if input_shape[-1] > 1:
/home/fxmarty/hf_internship/transformers/src/transformers/models/marian/modeling_marian.py:84: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  mask = torch.full((tgt_len, tgt_len), torch.tensor(torch.finfo(dtype).min))
2022-09-28 15:06:05.646069585 [I:onnxruntime:, inference_session.cc:262 operator()] Flush-to-zero and denormal-as-zero are off
2022-09-28 15:06:05.646094392 [I:onnxruntime:, inference_session.cc:270 ConstructorCommon] Creating and using per session threadpools since use_per_session_threads_ is true
2022-09-28 15:06:05.646100747 [I:onnxruntime:, inference_session.cc:291 ConstructorCommon] Dynamic block base set to 0
2022-09-28 15:06:07.295378832 [I:onnxruntime:, inference_session.cc:1246 Initialize] Initializing session.
2022-09-28 15:06:07.295403769 [I:onnxruntime:, inference_session.cc:1283 Initialize] Adding default CPU execution provider.
2022-09-28 15:06:07.302716431 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::ConstantOfShape_135'. It is no longer used by any node.
2022-09-28 15:06:07.302735827 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Mul_137'. It is no longer used by any node.
2022-09-28 15:06:07.302986539 [I:onnxruntime:, reshape_fusion.cc:38 ApplyImpl] Fused reshape node: input
2022-09-28 15:06:07.303064301 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 1
2022-09-28 15:06:07.305093486 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 0
2022-09-28 15:06:07.308666607 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_989'. It is no longer used by any node.
2022-09-28 15:06:07.308673050 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_986'. It is no longer used by any node.
2022-09-28 15:06:07.308676818 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_848'. It is no longer used by any node.
2022-09-28 15:06:07.308679324 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_828'. It is no longer used by any node.
2022-09-28 15:06:07.308682110 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_684'. It is no longer used by any node.
2022-09-28 15:06:07.308684970 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_543'. It is no longer used by any node.
2022-09-28 15:06:07.308687582 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_405'. It is no longer used by any node.
2022-09-28 15:06:07.308689969 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_402'. It is no longer used by any node.
2022-09-28 15:06:07.308693834 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_687'. It is no longer used by any node.
2022-09-28 15:06:07.308697700 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_264'. It is no longer used by any node.
2022-09-28 15:06:07.308700337 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_425'. It is no longer used by any node.
2022-09-28 15:06:07.308703487 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_845'. It is no longer used by any node.
2022-09-28 15:06:07.308706122 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_704'. It is no longer used by any node.
2022-09-28 15:06:07.308708977 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_825'. It is no longer used by any node.
2022-09-28 15:06:07.308712105 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_707'. It is no longer used by any node.
2022-09-28 15:06:07.308715451 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_566'. It is no longer used by any node.
2022-09-28 15:06:07.308718040 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_422'. It is no longer used by any node.
2022-09-28 15:06:07.308721108 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_563'. It is no longer used by any node.
2022-09-28 15:06:07.308723939 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_281'. It is no longer used by any node.
2022-09-28 15:06:07.308727067 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_546'. It is no longer used by any node.
2022-09-28 15:06:07.308729667 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_966'. It is no longer used by any node.
2022-09-28 15:06:07.308732612 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_969'. It is no longer used by any node.
2022-09-28 15:06:07.308735290 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_261'. It is no longer used by any node.
2022-09-28 15:06:07.308738037 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_284'. It is no longer used by any node.
2022-09-28 15:06:07.308912279 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.308916688 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.308919676 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.308922481 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.308925298 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.311135513 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.311142286 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.311146634 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.311150328 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.311153973 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.312280874 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.312284965 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.312287855 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.312290700 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.312293587 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.313413189 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.313418754 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.313422119 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.313425487 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.313428787 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.314445919 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.314449526 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.314452223 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.314454827 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.314457415 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.315469089 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.315474032 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.315477392 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.315480764 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.315484103 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.316624978 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.316629741 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.316632581 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.316635393 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.316638418 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.317779356 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.317783823 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.317786825 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.317789680 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.317792512 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.318842324 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.318846369 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.318849256 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.318852147 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.318855091 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.320068235 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.320072352 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.320075905 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.320079427 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.320082838 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:07.323264441 [V:onnxruntime:, session_state.cc:1186 VerifyEachNodeIsAssignedToAnEp] Node placements
2022-09-28 15:06:07.323275688 [V:onnxruntime:, session_state.cc:1193 VerifyEachNodeIsAssignedToAnEp]  Provider: [CPUExecutionProvider]: [Gather (Gather_107), Gather (Gather_120), Gather (Gather_123), Unsqueeze (Unsqueeze_126), Unsqueeze (Unsqueeze_128), Concat (Concat_129), Reshape (Reshape_131), Equal (Equal_136), Where (Where_137), Gather (Gather_148), Gather (Gather_151), Concat (Concat_159), Mul (Mul_169), Concat (Concat_172), Unsqueeze (Unsqueeze_178), Concat (Concat_179), Gather (Gather_186), Concat (Concat_192), Mul (Mul_196), Unsqueeze (Unsqueeze_197), Unsqueeze (Unsqueeze_199), Concat (Concat_200), Concat (Concat_206), Unsqueeze (Unsqueeze_209), Unsqueeze (Unsqueeze_210), Concat (Concat_211), Gather (Gather_247), Gather (Gather_250), Concat (Concat_258), Mul (Mul_268), Concat (Concat_271), Unsqueeze (Unsqueeze_277), Concat (Concat_278), Gather (Gather_285), Concat (Concat_291), Mul (Mul_295), Unsqueeze (Unsqueeze_296), Unsqueeze (Unsqueeze_298), Concat (Concat_299), Concat (Concat_305), Unsqueeze (Unsqueeze_308), Unsqueeze (Unsqueeze_309), Concat (Concat_310), Gather (Gather_346), Gather (Gather_349), Concat (Concat_357), Mul (Mul_367), Concat (Concat_370), Unsqueeze (Unsqueeze_376), Concat (Concat_377), Gather (Gather_384), Concat (Concat_390), Mul (Mul_394), Unsqueeze (Unsqueeze_395), Unsqueeze (Unsqueeze_397), Concat (Concat_398), Concat (Concat_404), Unsqueeze (Unsqueeze_407), Unsqueeze (Unsqueeze_408), Concat (Concat_409), Gather (Gather_445), Gather (Gather_448), Concat (Concat_456), Mul (Mul_466), Concat (Concat_469), Unsqueeze (Unsqueeze_475), Concat (Concat_476), Gather (Gather_483), Concat (Concat_489), Mul (Mul_493), Unsqueeze (Unsqueeze_494), Unsqueeze (Unsqueeze_496), Concat (Concat_497), Concat (Concat_503), Unsqueeze (Unsqueeze_506), Unsqueeze (Unsqueeze_507), Concat (Concat_508), Gather (Gather_544), Gather (Gather_547), Concat (Concat_555), Mul (Mul_565), Concat (Concat_568), Unsqueeze (Unsqueeze_574), Concat (Concat_575), Gather (Gather_582), Concat (Concat_588), Mul (Mul_592), Unsqueeze (Unsqueeze_593), Unsqueeze (Unsqueeze_595), Concat (Concat_596), Concat (Concat_602), Unsqueeze (Unsqueeze_605), Unsqueeze (Unsqueeze_606), Concat (Concat_607), Gather (Gather_643), Gather (Gather_646), Concat (Concat_654), Mul (Mul_664), Concat (Concat_667), Unsqueeze (Unsqueeze_673), Concat (Concat_674), Gather (Gather_681), Concat (Concat_687), Mul (Mul_691), Unsqueeze (Unsqueeze_692), Unsqueeze (Unsqueeze_694), Concat (Concat_695), Concat (Concat_701), Unsqueeze (Unsqueeze_704), Unsqueeze (Unsqueeze_705), Concat (Concat_706), ]
2022-09-28 15:06:07.323292712 [V:onnxruntime:, session_state.cc:1193 VerifyEachNodeIsAssignedToAnEp]  Provider: [CUDAExecutionProvider]: [Shape (Shape_105), Reshape (Reshape_110), Gather (Gather_111), Mul (Mul_113), Range (Range_115), Gather (Gather_116), Add (Add_117), Shape (Shape_121), Unsqueeze (Unsqueeze_124), Unsqueeze (Unsqueeze_125), Expand (Expand_138), Cast (Cast_139), Sub (Sub_141), Cast (Cast_142), Where (Where_145), Shape (Shape_149), MatMul (MatMul_152), Add (Add_153), Mul (Mul_155), MatMul (MatMul_156), Add (Add_157), Reshape (Reshape_160), Transpose (Transpose_161), MatMul (MatMul_162), Add (Add_163), Reshape (Reshape_166), Transpose (Transpose_167), Reshape (Reshape_173), Transpose (Transpose_174), Reshape (Reshape_177), Reshape (Reshape_180), Reshape (Reshape_183), Shape (Shape_184), Reshape (Reshape_193), Add (Add_194), Reshape (Reshape_201), Softmax (Softmax_202), MatMul (MatMul_203), Reshape (Reshape_207), Transpose (Transpose_208), Reshape (Reshape_212), MatMul (MatMul_213), Add (Add_214), Add (Add_215), MatMul (MatMul_227), Add (Add_228), Sigmoid (Sigmoid_229), Mul (Mul_230), MatMul (MatMul_231), Add (Add_232), Add (Add_233), Shape (Shape_248), MatMul (MatMul_251), Add (Add_252), Mul (Mul_254), MatMul (MatMul_255), Add (Add_256), Reshape (Reshape_259), Transpose (Transpose_260), MatMul (MatMul_261), Add (Add_262), Reshape (Reshape_265), Transpose (Transpose_266), Reshape (Reshape_272), Transpose (Transpose_273), Reshape (Reshape_276), Reshape (Reshape_279), Reshape (Reshape_282), Shape (Shape_283), Reshape (Reshape_292), Add (Add_293), Reshape (Reshape_300), Softmax (Softmax_301), MatMul (MatMul_302), Reshape (Reshape_306), Transpose (Transpose_307), Reshape (Reshape_311), MatMul (MatMul_312), Add (Add_313), Add (Add_314), MatMul (MatMul_326), Add (Add_327), Sigmoid (Sigmoid_328), Mul (Mul_329), MatMul (MatMul_330), Add (Add_331), Add (Add_332), Shape (Shape_347), MatMul (MatMul_350), Add (Add_351), Mul (Mul_353), MatMul (MatMul_354), Add (Add_355), Reshape (Reshape_358), Transpose (Transpose_359), MatMul (MatMul_360), Add (Add_361), Reshape (Reshape_364), Transpose (Transpose_365), Reshape (Reshape_371), Transpose (Transpose_372), Reshape (Reshape_375), Reshape (Reshape_378), Reshape (Reshape_381), Shape (Shape_382), Reshape (Reshape_391), Add (Add_392), Reshape (Reshape_399), Softmax (Softmax_400), MatMul (MatMul_401), Reshape (Reshape_405), Transpose (Transpose_406), Reshape (Reshape_410), MatMul (MatMul_411), Add (Add_412), Add (Add_413), MatMul (MatMul_425), Add (Add_426), Sigmoid (Sigmoid_427), Mul (Mul_428), MatMul (MatMul_429), Add (Add_430), Add (Add_431), Shape (Shape_446), MatMul (MatMul_449), Add (Add_450), Mul (Mul_452), MatMul (MatMul_453), Add (Add_454), Reshape (Reshape_457), Transpose (Transpose_458), MatMul (MatMul_459), Add (Add_460), Reshape (Reshape_463), Transpose (Transpose_464), Reshape (Reshape_470), Transpose (Transpose_471), Reshape (Reshape_474), Reshape (Reshape_477), Reshape (Reshape_480), Shape (Shape_481), Reshape (Reshape_490), Add (Add_491), Reshape (Reshape_498), Softmax (Softmax_499), MatMul (MatMul_500), Reshape (Reshape_504), Transpose (Transpose_505), Reshape (Reshape_509), MatMul (MatMul_510), Add (Add_511), Add (Add_512), MatMul (MatMul_524), Add (Add_525), Sigmoid (Sigmoid_526), Mul (Mul_527), MatMul (MatMul_528), Add (Add_529), Add (Add_530), Shape (Shape_545), MatMul (MatMul_548), Add (Add_549), Mul (Mul_551), MatMul (MatMul_552), Add (Add_553), Reshape (Reshape_556), Transpose (Transpose_557), MatMul (MatMul_558), Add (Add_559), Reshape (Reshape_562), Transpose (Transpose_563), Reshape (Reshape_569), Transpose (Transpose_570), Reshape (Reshape_573), Reshape (Reshape_576), Reshape (Reshape_579), Shape (Shape_580), Reshape (Reshape_589), Add (Add_590), Reshape (Reshape_597), Softmax (Softmax_598), MatMul (MatMul_599), Reshape (Reshape_603), Transpose (Transpose_604), Reshape (Reshape_608), MatMul (MatMul_609), Add (Add_610), Add (Add_611), MatMul (MatMul_623), Add (Add_624), Sigmoid (Sigmoid_625), Mul (Mul_626), MatMul (MatMul_627), Add (Add_628), Add (Add_629), Shape (Shape_644), MatMul (MatMul_647), Add (Add_648), Mul (Mul_650), MatMul (MatMul_651), Add (Add_652), Reshape (Reshape_655), Transpose (Transpose_656), MatMul (MatMul_657), Add (Add_658), Reshape (Reshape_661), Transpose (Transpose_662), Reshape (Reshape_668), Transpose (Transpose_669), Reshape (Reshape_672), Reshape (Reshape_675), Reshape (Reshape_678), Shape (Shape_679), Reshape (Reshape_688), Add (Add_689), Reshape (Reshape_696), Softmax (Softmax_697), MatMul (MatMul_698), Reshape (Reshape_702), Transpose (Transpose_703), Reshape (Reshape_707), MatMul (MatMul_708), Add (Add_709), Add (Add_710), MatMul (MatMul_722), Add (Add_723), Sigmoid (Sigmoid_724), Mul (Mul_725), MatMul (MatMul_726), Add (Add_727), Add (Add_728), LayerNormalization (LayerNormalization), LayerNormalization (LayerNormalization_token_0), LayerNormalization (LayerNormalization_token_1), LayerNormalization (LayerNormalization_token_2), LayerNormalization (LayerNormalization_token_3), LayerNormalization (LayerNormalization_token_4), LayerNormalization (LayerNormalization_token_5), LayerNormalization (LayerNormalization_token_6), LayerNormalization (LayerNormalization_token_7), LayerNormalization (LayerNormalization_token_8), LayerNormalization (LayerNormalization_token_9), LayerNormalization (LayerNormalization_token_10), FusedMatMul (MatMul_With_Transpose), FusedMatMul (MatMul_With_Transpose_token_11), FusedMatMul (MatMul_With_Transpose_token_12), FusedMatMul (MatMul_With_Transpose_token_13), FusedMatMul (MatMul_With_Transpose_token_14), FusedMatMul (MatMul_With_Transpose_token_15), ]
2022-09-28 15:06:07.323885265 [V:onnxruntime:, session_state.cc:81 CreateGraphInfo] SaveMLValueNameIndexMapping
2022-09-28 15:06:07.323943993 [V:onnxruntime:, session_state.cc:127 CreateGraphInfo] Done saving OrtValue mappings.
2022-09-28 15:06:12.027641039 [I:onnxruntime:, session_state_utils.cc:140 SaveInitializedTensors] Saving initialized tensors.
2022-09-28 15:06:12.296239742 [I:onnxruntime:, session_state_utils.cc:268 SaveInitializedTensors] Done saving initialized tensors
2022-09-28 15:06:12.300683506 [I:onnxruntime:, inference_session.cc:1518 Initialize] Session successfully initialized.
2022-09-28 15:06:12.300811319 [I:onnxruntime:, inference_session.cc:270 ConstructorCommon] Creating and using per session threadpools since use_per_session_threads_ is true
2022-09-28 15:06:12.300821246 [I:onnxruntime:, inference_session.cc:291 ConstructorCommon] Dynamic block base set to 0
2022-09-28 15:06:12.552977060 [I:onnxruntime:, inference_session.cc:1246 Initialize] Initializing session.
2022-09-28 15:06:12.552994602 [I:onnxruntime:, inference_session.cc:1283 Initialize] Adding default CPU execution provider.
2022-09-28 15:06:12.563600851 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::ConstantOfShape_241'. It is no longer used by any node.
2022-09-28 15:06:12.563625585 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::ConstantOfShape_217'. It is no longer used by any node.
2022-09-28 15:06:12.563648605 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Mul_243'. It is no longer used by any node.
2022-09-28 15:06:12.563656922 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Mul_219'. It is no longer used by any node.
2022-09-28 15:06:12.564636464 [I:onnxruntime:, reshape_fusion.cc:38 ApplyImpl] Fused reshape node: input
2022-09-28 15:06:12.564687490 [I:onnxruntime:, reshape_fusion.cc:38 ApplyImpl] Fused reshape node: onnx::Less_201
2022-09-28 15:06:12.565069876 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 2
2022-09-28 15:06:12.570538590 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Slice_192'. It is no longer used by any node.
2022-09-28 15:06:12.570574082 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Slice_193'. It is no longer used by any node.
2022-09-28 15:06:12.570582284 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Slice_194'. It is no longer used by any node.
2022-09-28 15:06:12.577215280 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 0
2022-09-28 15:06:12.596488214 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1829'. It is no longer used by any node.
2022-09-28 15:06:12.596520733 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1809'. It is no longer used by any node.
2022-09-28 15:06:12.596531371 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1688'. It is no longer used by any node.
2022-09-28 15:06:12.596537923 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1685'. It is no longer used by any node.
2022-09-28 15:06:12.596543905 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1567'. It is no longer used by any node.
2022-09-28 15:06:12.596549608 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1564'. It is no longer used by any node.
2022-09-28 15:06:12.596555383 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1544'. It is no longer used by any node.
2022-09-28 15:06:12.596561421 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1426'. It is no longer used by any node.
2022-09-28 15:06:12.596567451 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1423'. It is no longer used by any node.
2022-09-28 15:06:12.596574940 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1020'. It is no longer used by any node.
2022-09-28 15:06:12.596580802 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1826'. It is no longer used by any node.
2022-09-28 15:06:12.596587020 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1305'. It is no longer used by any node.
2022-09-28 15:06:12.596593965 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_640'. It is no longer used by any node.
2022-09-28 15:06:12.596599646 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1547'. It is no longer used by any node.
2022-09-28 15:06:12.596605698 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_761'. It is no longer used by any node.
2022-09-28 15:06:12.596616255 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_758'. It is no longer used by any node.
2022-09-28 15:06:12.596625691 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1161'. It is no longer used by any node.
2022-09-28 15:06:12.596632660 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1285'. It is no longer used by any node.
2022-09-28 15:06:12.596640105 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1040'. It is no longer used by any node.
2022-09-28 15:06:12.596647352 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1302'. It is no longer used by any node.
2022-09-28 15:06:12.596653671 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_778'. It is no longer used by any node.
2022-09-28 15:06:12.596661233 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_378'. It is no longer used by any node.
2022-09-28 15:06:12.596666864 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1806'. It is no longer used by any node.
2022-09-28 15:06:12.596675337 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_637'. It is no longer used by any node.
2022-09-28 15:06:12.596680840 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_496'. It is no longer used by any node.
2022-09-28 15:06:12.596687712 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_899'. It is no longer used by any node.
2022-09-28 15:06:12.596694194 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1023'. It is no longer used by any node.
2022-09-28 15:06:12.596700663 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1282'. It is no longer used by any node.
2022-09-28 15:06:12.596709322 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1043'. It is no longer used by any node.
2022-09-28 15:06:12.596716813 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1164'. It is no longer used by any node.
2022-09-28 15:06:12.596723422 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_375'. It is no longer used by any node.
2022-09-28 15:06:12.596729486 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_499'. It is no longer used by any node.
2022-09-28 15:06:12.596734925 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_516'. It is no longer used by any node.
2022-09-28 15:06:12.596740654 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_519'. It is no longer used by any node.
2022-09-28 15:06:12.596747081 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_781'. It is no longer used by any node.
2022-09-28 15:06:12.596754448 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_902'. It is no longer used by any node.
2022-09-28 15:06:12.597479914 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.597494397 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.597502710 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.597512494 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.597518995 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.607565178 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.607604933 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.607613780 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.607622765 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.607630545 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.611525820 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.611546746 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.611555147 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.611562152 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.611568726 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.615111460 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.615126919 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.615133807 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.615139936 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.615146388 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.618571521 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.618584036 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.618589885 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.618595260 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.618601949 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.621755935 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.621771257 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.621776994 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.621782359 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.621787964 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.624819018 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.624833191 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.624838404 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.624843467 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.624848722 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.627666271 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.627674989 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.627680019 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.627685359 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.627690137 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.630364099 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.630372636 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.630377342 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.630381838 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.630386416 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.632902760 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.632909984 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.632913866 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.632919473 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.632923724 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:12.642598171 [V:onnxruntime:, session_state.cc:1186 VerifyEachNodeIsAssignedToAnEp] Node placements
2022-09-28 15:06:12.642630162 [V:onnxruntime:, session_state.cc:1193 VerifyEachNodeIsAssignedToAnEp]  Provider: [CUDAExecutionProvider]: [Shape (Shape_218), Reshape (Reshape_223), Gather (Gather_224), Mul (Mul_226), ConstantOfShape (ConstantOfShape_230), Range (Range_240), Add (Add_242), Shape (Shape_243), Reshape (Reshape_251), Less (Less_252), Where (Where_255), Unsqueeze (Unsqueeze_257), Unsqueeze (Unsqueeze_258), Expand (Expand_271), Shape (Shape_276), Unsqueeze (Unsqueeze_279), Unsqueeze (Unsqueeze_280), Expand (Expand_293), Cast (Cast_294), Sub (Sub_296), Cast (Cast_297), Where (Where_300), Range (Range_302), Gather (Gather_303), Add (Add_304), Shape (Shape_308), MatMul (MatMul_311), Add (Add_312), Mul (Mul_314), MatMul (MatMul_315), Add (Add_316), Reshape (Reshape_319), Transpose (Transpose_320), MatMul (MatMul_321), Add (Add_322), Reshape (Reshape_325), Transpose (Transpose_326), Reshape (Reshape_332), Transpose (Transpose_333), Reshape (Reshape_336), Reshape (Reshape_339), Reshape (Reshape_342), Shape (Shape_343), Reshape (Reshape_352), Add (Add_353), Reshape (Reshape_360), Softmax (Softmax_361), MatMul (MatMul_362), Reshape (Reshape_366), Transpose (Transpose_367), Reshape (Reshape_371), MatMul (MatMul_372), Add (Add_373), Add (Add_374), Shape (Shape_389), MatMul (MatMul_392), Add (Add_393), Mul (Mul_395), MatMul (MatMul_396), Add (Add_397), Reshape (Reshape_400), Transpose (Transpose_401), MatMul (MatMul_402), Add (Add_403), Reshape (Reshape_406), Transpose (Transpose_407), Reshape (Reshape_413), Transpose (Transpose_414), Reshape (Reshape_417), Reshape (Reshape_420), Reshape (Reshape_423), Shape (Shape_424), Reshape (Reshape_433), Add (Add_434), Reshape (Reshape_441), Softmax (Softmax_442), MatMul (MatMul_443), Reshape (Reshape_447), Transpose (Transpose_448), Reshape (Reshape_452), MatMul (MatMul_453), Add (Add_454), Add (Add_455), MatMul (MatMul_467), Add (Add_468), Sigmoid (Sigmoid_469), Mul (Mul_470), MatMul (MatMul_471), Add (Add_472), Add (Add_473), Shape (Shape_488), MatMul (MatMul_491), Add (Add_492), Mul (Mul_494), MatMul (MatMul_495), Add (Add_496), Reshape (Reshape_499), Transpose (Transpose_500), MatMul (MatMul_501), Add (Add_502), Reshape (Reshape_505), Transpose (Transpose_506), Reshape (Reshape_512), Transpose (Transpose_513), Reshape (Reshape_516), Reshape (Reshape_519), Reshape (Reshape_522), Shape (Shape_523), Reshape (Reshape_532), Add (Add_533), Reshape (Reshape_540), Softmax (Softmax_541), MatMul (MatMul_542), Reshape (Reshape_546), Transpose (Transpose_547), Reshape (Reshape_551), MatMul (MatMul_552), Add (Add_553), Add (Add_554), Shape (Shape_569), MatMul (MatMul_572), Add (Add_573), Mul (Mul_575), MatMul (MatMul_576), Add (Add_577), Reshape (Reshape_580), Transpose (Transpose_581), MatMul (MatMul_582), Add (Add_583), Reshape (Reshape_586), Transpose (Transpose_587), Reshape (Reshape_593), Transpose (Transpose_594), Reshape (Reshape_597), Reshape (Reshape_600), Reshape (Reshape_603), Shape (Shape_604), Reshape (Reshape_613), Add (Add_614), Reshape (Reshape_621), Softmax (Softmax_622), MatMul (MatMul_623), Reshape (Reshape_627), Transpose (Transpose_628), Reshape (Reshape_632), MatMul (MatMul_633), Add (Add_634), Add (Add_635), MatMul (MatMul_647), Add (Add_648), Sigmoid (Sigmoid_649), Mul (Mul_650), MatMul (MatMul_651), Add (Add_652), Add (Add_653), Shape (Shape_668), MatMul (MatMul_671), Add (Add_672), Mul (Mul_674), MatMul (MatMul_675), Add (Add_676), Reshape (Reshape_679), Transpose (Transpose_680), MatMul (MatMul_681), Add (Add_682), Reshape (Reshape_685), Transpose (Transpose_686), Reshape (Reshape_692), Transpose (Transpose_693), Reshape (Reshape_696), Reshape (Reshape_699), Reshape (Reshape_702), Shape (Shape_703), Reshape (Reshape_712), Add (Add_713), Reshape (Reshape_720), Softmax (Softmax_721), MatMul (MatMul_722), Reshape (Reshape_726), Transpose (Transpose_727), Reshape (Reshape_731), MatMul (MatMul_732), Add (Add_733), Add (Add_734), Shape (Shape_749), MatMul (MatMul_752), Add (Add_753), Mul (Mul_755), MatMul (MatMul_756), Add (Add_757), Reshape (Reshape_760), Transpose (Transpose_761), MatMul (MatMul_762), Add (Add_763), Reshape (Reshape_766), Transpose (Transpose_767), Reshape (Reshape_773), Transpose (Transpose_774), Reshape (Reshape_777), Reshape (Reshape_780), Reshape (Reshape_783), Shape (Shape_784), Reshape (Reshape_793), Add (Add_794), Reshape (Reshape_801), Softmax (Softmax_802), MatMul (MatMul_803), Reshape (Reshape_807), Transpose (Transpose_808), Reshape (Reshape_812), MatMul (MatMul_813), Add (Add_814), Add (Add_815), MatMul (MatMul_827), Add (Add_828), Sigmoid (Sigmoid_829), Mul (Mul_830), MatMul (MatMul_831), Add (Add_832), Add (Add_833), Shape (Shape_848), MatMul (MatMul_851), Add (Add_852), Mul (Mul_854), MatMul (MatMul_855), Add (Add_856), Reshape (Reshape_859), Transpose (Transpose_860), MatMul (MatMul_861), Add (Add_862), Reshape (Reshape_865), Transpose (Transpose_866), Reshape (Reshape_872), Transpose (Transpose_873), Reshape (Reshape_876), Reshape (Reshape_879), Reshape (Reshape_882), Shape (Shape_883), Reshape (Reshape_892), Add (Add_893), Reshape (Reshape_900), Softmax (Softmax_901), MatMul (MatMul_902), Reshape (Reshape_906), Transpose (Transpose_907), Reshape (Reshape_911), MatMul (MatMul_912), Add (Add_913), Add (Add_914), Shape (Shape_929), MatMul (MatMul_932), Add (Add_933), Mul (Mul_935), MatMul (MatMul_936), Add (Add_937), Reshape (Reshape_940), Transpose (Transpose_941), MatMul (MatMul_942), Add (Add_943), Reshape (Reshape_946), Transpose (Transpose_947), Reshape (Reshape_953), Transpose (Transpose_954), Reshape (Reshape_957), Reshape (Reshape_960), Reshape (Reshape_963), Shape (Shape_964), Reshape (Reshape_973), Add (Add_974), Reshape (Reshape_981), Softmax (Softmax_982), MatMul (MatMul_983), Reshape (Reshape_987), Transpose (Transpose_988), Reshape (Reshape_992), MatMul (MatMul_993), Add (Add_994), Add (Add_995), MatMul (MatMul_1007), Add (Add_1008), Sigmoid (Sigmoid_1009), Mul (Mul_1010), MatMul (MatMul_1011), Add (Add_1012), Add (Add_1013), Shape (Shape_1028), MatMul (MatMul_1031), Add (Add_1032), Mul (Mul_1034), MatMul (MatMul_1035), Add (Add_1036), Reshape (Reshape_1039), Transpose (Transpose_1040), MatMul (MatMul_1041), Add (Add_1042), Reshape (Reshape_1045), Transpose (Transpose_1046), Reshape (Reshape_1052), Transpose (Transpose_1053), Reshape (Reshape_1056), Reshape (Reshape_1059), Reshape (Reshape_1062), Shape (Shape_1063), Reshape (Reshape_1072), Add (Add_1073), Reshape (Reshape_1080), Softmax (Softmax_1081), MatMul (MatMul_1082), Reshape (Reshape_1086), Transpose (Transpose_1087), Reshape (Reshape_1091), MatMul (MatMul_1092), Add (Add_1093), Add (Add_1094), Shape (Shape_1109), MatMul (MatMul_1112), Add (Add_1113), Mul (Mul_1115), MatMul (MatMul_1116), Add (Add_1117), Reshape (Reshape_1120), Transpose (Transpose_1121), MatMul (MatMul_1122), Add (Add_1123), Reshape (Reshape_1126), Transpose (Transpose_1127), Reshape (Reshape_1133), Transpose (Transpose_1134), Reshape (Reshape_1137), Reshape (Reshape_1140), Reshape (Reshape_1143), Shape (Shape_1144), Reshape (Reshape_1153), Add (Add_1154), Reshape (Reshape_1161), Softmax (Softmax_1162), MatMul (MatMul_1163), Reshape (Reshape_1167), Transpose (Transpose_1168), Reshape (Reshape_1172), MatMul (MatMul_1173), Add (Add_1174), Add (Add_1175), MatMul (MatMul_1187), Add (Add_1188), Sigmoid (Sigmoid_1189), Mul (Mul_1190), MatMul (MatMul_1191), Add (Add_1192), Add (Add_1193), Shape (Shape_1208), MatMul (MatMul_1211), Add (Add_1212), Mul (Mul_1214), MatMul (MatMul_1215), Add (Add_1216), Reshape (Reshape_1219), Transpose (Transpose_1220), MatMul (MatMul_1221), Add (Add_1222), Reshape (Reshape_1225), Transpose (Transpose_1226), Reshape (Reshape_1232), Transpose (Transpose_1233), Reshape (Reshape_1236), Reshape (Reshape_1239), Reshape (Reshape_1242), Shape (Shape_1243), Reshape (Reshape_1252), Add (Add_1253), Reshape (Reshape_1260), Softmax (Softmax_1261), MatMul (MatMul_1262), Reshape (Reshape_1266), Transpose (Transpose_1267), Reshape (Reshape_1271), MatMul (MatMul_1272), Add (Add_1273), Add (Add_1274), Shape (Shape_1289), MatMul (MatMul_1292), Add (Add_1293), Mul (Mul_1295), MatMul (MatMul_1296), Add (Add_1297), Reshape (Reshape_1300), Transpose (Transpose_1301), MatMul (MatMul_1302), Add (Add_1303), Reshape (Reshape_1306), Transpose (Transpose_1307), Reshape (Reshape_1313), Transpose (Transpose_1314), Reshape (Reshape_1317), Reshape (Reshape_1320), Reshape (Reshape_1323), Shape (Shape_1324), Reshape (Reshape_1333), Add (Add_1334), Reshape (Reshape_1341), Softmax (Softmax_1342), MatMul (MatMul_1343), Reshape (Reshape_1347), Transpose (Transpose_1348), Reshape (Reshape_1352), MatMul (MatMul_1353), Add (Add_1354), Add (Add_1355), MatMul (MatMul_1367), Add (Add_1368), Sigmoid (Sigmoid_1369), Mul (Mul_1370), MatMul (MatMul_1371), Add (Add_1372), Add (Add_1373), MatMul (MatMul_1385), Add (Add_1387), LayerNormalization (LayerNormalization), LayerNormalization (LayerNormalization_token_0), LayerNormalization (LayerNormalization_token_1), LayerNormalization (LayerNormalization_token_2), LayerNormalization (LayerNormalization_token_3), LayerNormalization (LayerNormalization_token_4), LayerNormalization (LayerNormalization_token_5), LayerNormalization (LayerNormalization_token_6), LayerNormalization (LayerNormalization_token_7), LayerNormalization (LayerNormalization_token_8), LayerNormalization (LayerNormalization_token_9), LayerNormalization (LayerNormalization_token_10), LayerNormalization (LayerNormalization_token_11), LayerNormalization (LayerNormalization_token_12), LayerNormalization (LayerNormalization_token_13), LayerNormalization (LayerNormalization_token_14), LayerNormalization (LayerNormalization_token_15), LayerNormalization (LayerNormalization_token_16), FusedMatMul (MatMul_With_Transpose), FusedMatMul (MatMul_With_Transpose_token_17), FusedMatMul (MatMul_With_Transpose_token_18), FusedMatMul (MatMul_With_Transpose_token_19), FusedMatMul (MatMul_With_Transpose_token_20), FusedMatMul (MatMul_With_Transpose_token_21), FusedMatMul (MatMul_With_Transpose_token_22), FusedMatMul (MatMul_With_Transpose_token_23), FusedMatMul (MatMul_With_Transpose_token_24), FusedMatMul (MatMul_With_Transpose_token_25), FusedMatMul (MatMul_With_Transpose_token_26), FusedMatMul (MatMul_With_Transpose_token_27), ]
2022-09-28 15:06:12.642884335 [V:onnxruntime:, session_state.cc:1193 VerifyEachNodeIsAssignedToAnEp]  Provider: [CPUExecutionProvider]: [Gather (Gather_217), Gather (Gather_220), Unsqueeze (Unsqueeze_221), Concat (Concat_229), Slice (Slice_235), Squeeze (Squeeze_236), Unsqueeze (Unsqueeze_259), Concat (Concat_262), Reshape (Reshape_264), Equal (Equal_269), Where (Where_270), Gather (Gather_275), Gather (Gather_278), Unsqueeze (Unsqueeze_281), Unsqueeze (Unsqueeze_283), Concat (Concat_284), Reshape (Reshape_286), Equal (Equal_291), Where (Where_292), Gather (Gather_307), Gather (Gather_310), Concat (Concat_318), Mul (Mul_328), Concat (Concat_331), Unsqueeze (Unsqueeze_337), Concat (Concat_338), Gather (Gather_345), Concat (Concat_351), Mul (Mul_355), Unsqueeze (Unsqueeze_356), Unsqueeze (Unsqueeze_358), Concat (Concat_359), Concat (Concat_365), Unsqueeze (Unsqueeze_368), Unsqueeze (Unsqueeze_369), Concat (Concat_370), Gather (Gather_388), Gather (Gather_391), Concat (Concat_399), Mul (Mul_409), Concat (Concat_412), Unsqueeze (Unsqueeze_418), Concat (Concat_419), Gather (Gather_426), Concat (Concat_432), Mul (Mul_436), Unsqueeze (Unsqueeze_437), Unsqueeze (Unsqueeze_439), Concat (Concat_440), Concat (Concat_446), Unsqueeze (Unsqueeze_449), Unsqueeze (Unsqueeze_450), Concat (Concat_451), Gather (Gather_487), Gather (Gather_490), Concat (Concat_498), Mul (Mul_508), Concat (Concat_511), Unsqueeze (Unsqueeze_517), Concat (Concat_518), Gather (Gather_525), Concat (Concat_531), Mul (Mul_535), Unsqueeze (Unsqueeze_536), Unsqueeze (Unsqueeze_538), Concat (Concat_539), Concat (Concat_545), Unsqueeze (Unsqueeze_548), Unsqueeze (Unsqueeze_549), Concat (Concat_550), Gather (Gather_568), Gather (Gather_571), Concat (Concat_579), Mul (Mul_589), Concat (Concat_592), Unsqueeze (Unsqueeze_598), Concat (Concat_599), Gather (Gather_606), Concat (Concat_612), Mul (Mul_616), Unsqueeze (Unsqueeze_617), Unsqueeze (Unsqueeze_619), Concat (Concat_620), Concat (Concat_626), Unsqueeze (Unsqueeze_629), Unsqueeze (Unsqueeze_630), Concat (Concat_631), Gather (Gather_667), Gather (Gather_670), Concat (Concat_678), Mul (Mul_688), Concat (Concat_691), Unsqueeze (Unsqueeze_697), Concat (Concat_698), Gather (Gather_705), Concat (Concat_711), Mul (Mul_715), Unsqueeze (Unsqueeze_716), Unsqueeze (Unsqueeze_718), Concat (Concat_719), Concat (Concat_725), Unsqueeze (Unsqueeze_728), Unsqueeze (Unsqueeze_729), Concat (Concat_730), Gather (Gather_748), Gather (Gather_751), Concat (Concat_759), Mul (Mul_769), Concat (Concat_772), Unsqueeze (Unsqueeze_778), Concat (Concat_779), Gather (Gather_786), Concat (Concat_792), Mul (Mul_796), Unsqueeze (Unsqueeze_797), Unsqueeze (Unsqueeze_799), Concat (Concat_800), Concat (Concat_806), Unsqueeze (Unsqueeze_809), Unsqueeze (Unsqueeze_810), Concat (Concat_811), Gather (Gather_847), Gather (Gather_850), Concat (Concat_858), Mul (Mul_868), Concat (Concat_871), Unsqueeze (Unsqueeze_877), Concat (Concat_878), Gather (Gather_885), Concat (Concat_891), Mul (Mul_895), Unsqueeze (Unsqueeze_896), Unsqueeze (Unsqueeze_898), Concat (Concat_899), Concat (Concat_905), Unsqueeze (Unsqueeze_908), Unsqueeze (Unsqueeze_909), Concat (Concat_910), Gather (Gather_928), Gather (Gather_931), Concat (Concat_939), Mul (Mul_949), Concat (Concat_952), Unsqueeze (Unsqueeze_958), Concat (Concat_959), Gather (Gather_966), Concat (Concat_972), Mul (Mul_976), Unsqueeze (Unsqueeze_977), Unsqueeze (Unsqueeze_979), Concat (Concat_980), Concat (Concat_986), Unsqueeze (Unsqueeze_989), Unsqueeze (Unsqueeze_990), Concat (Concat_991), Gather (Gather_1027), Gather (Gather_1030), Concat (Concat_1038), Mul (Mul_1048), Concat (Concat_1051), Unsqueeze (Unsqueeze_1057), Concat (Concat_1058), Gather (Gather_1065), Concat (Concat_1071), Mul (Mul_1075), Unsqueeze (Unsqueeze_1076), Unsqueeze (Unsqueeze_1078), Concat (Concat_1079), Concat (Concat_1085), Unsqueeze (Unsqueeze_1088), Unsqueeze (Unsqueeze_1089), Concat (Concat_1090), Gather (Gather_1108), Gather (Gather_1111), Concat (Concat_1119), Mul (Mul_1129), Concat (Concat_1132), Unsqueeze (Unsqueeze_1138), Concat (Concat_1139), Gather (Gather_1146), Concat (Concat_1152), Mul (Mul_1156), Unsqueeze (Unsqueeze_1157), Unsqueeze (Unsqueeze_1159), Concat (Concat_1160), Concat (Concat_1166), Unsqueeze (Unsqueeze_1169), Unsqueeze (Unsqueeze_1170), Concat (Concat_1171), Gather (Gather_1207), Gather (Gather_1210), Concat (Concat_1218), Mul (Mul_1228), Concat (Concat_1231), Unsqueeze (Unsqueeze_1237), Concat (Concat_1238), Gather (Gather_1245), Concat (Concat_1251), Mul (Mul_1255), Unsqueeze (Unsqueeze_1256), Unsqueeze (Unsqueeze_1258), Concat (Concat_1259), Concat (Concat_1265), Unsqueeze (Unsqueeze_1268), Unsqueeze (Unsqueeze_1269), Concat (Concat_1270), Gather (Gather_1288), Gather (Gather_1291), Concat (Concat_1299), Mul (Mul_1309), Concat (Concat_1312), Unsqueeze (Unsqueeze_1318), Concat (Concat_1319), Gather (Gather_1326), Concat (Concat_1332), Mul (Mul_1336), Unsqueeze (Unsqueeze_1337), Unsqueeze (Unsqueeze_1339), Concat (Concat_1340), Concat (Concat_1346), Unsqueeze (Unsqueeze_1349), Unsqueeze (Unsqueeze_1350), Concat (Concat_1351), ]
2022-09-28 15:06:12.644230631 [V:onnxruntime:, session_state.cc:81 CreateGraphInfo] SaveMLValueNameIndexMapping
2022-09-28 15:06:12.644372686 [V:onnxruntime:, session_state.cc:127 CreateGraphInfo] Done saving OrtValue mappings.
2022-09-28 15:06:12.646371386 [I:onnxruntime:, session_state_utils.cc:140 SaveInitializedTensors] Saving initialized tensors.
2022-09-28 15:06:12.795362737 [I:onnxruntime:, session_state_utils.cc:268 SaveInitializedTensors] Done saving initialized tensors
2022-09-28 15:06:12.803549585 [I:onnxruntime:, inference_session.cc:1518 Initialize] Session successfully initialized.
2022-09-28 15:06:12.803710401 [I:onnxruntime:, inference_session.cc:270 ConstructorCommon] Creating and using per session threadpools since use_per_session_threads_ is true
2022-09-28 15:06:12.803719178 [I:onnxruntime:, inference_session.cc:291 ConstructorCommon] Dynamic block base set to 0
2022-09-28 15:06:13.153974853 [I:onnxruntime:, inference_session.cc:1246 Initialize] Initializing session.
2022-09-28 15:06:13.154015466 [I:onnxruntime:, inference_session.cc:1283 Initialize] Adding default CPU execution provider.
2022-09-28 15:06:13.164791422 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::ConstantOfShape_215'. It is no longer used by any node.
2022-09-28 15:06:13.164834961 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Mul_217'. It is no longer used by any node.
2022-09-28 15:06:13.165296871 [I:onnxruntime:, reshape_fusion.cc:38 ApplyImpl] Fused reshape node: input
2022-09-28 15:06:13.165465205 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 1
2022-09-28 15:06:13.170374576 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 0
2022-09-28 15:06:13.179235590 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1528'. It is no longer used by any node.
2022-09-28 15:06:13.179257101 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1331'. It is no longer used by any node.
2022-09-28 15:06:13.179262885 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1308'. It is no longer used by any node.
2022-09-28 15:06:13.179266861 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1216'. It is no longer used by any node.
2022-09-28 15:06:13.179270103 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1111'. It is no longer used by any node.
2022-09-28 15:06:13.179273787 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1108'. It is no longer used by any node.
2022-09-28 15:06:13.179277025 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1088'. It is no longer used by any node.
2022-09-28 15:06:13.179280306 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_996'. It is no longer used by any node.
2022-09-28 15:06:13.179283670 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_993'. It is no longer used by any node.
2022-09-28 15:06:13.179287171 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_888'. It is no longer used by any node.
2022-09-28 15:06:13.179290441 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_871'. It is no longer used by any node.
2022-09-28 15:06:13.179294277 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_773'. It is no longer used by any node.
2022-09-28 15:06:13.179298434 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_891'. It is no longer used by any node.
2022-09-28 15:06:13.179301723 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_671'. It is no longer used by any node.
2022-09-28 15:06:13.179307392 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_648'. It is no longer used by any node.
2022-09-28 15:06:13.179311170 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_556'. It is no longer used by any node.
2022-09-28 15:06:13.179314557 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_553'. It is no longer used by any node.
2022-09-28 15:06:13.179318621 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_451'. It is no longer used by any node.
2022-09-28 15:06:13.179324258 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_448'. It is no longer used by any node.
2022-09-28 15:06:13.179327651 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_428'. It is no longer used by any node.
2022-09-28 15:06:13.179331318 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_336'. It is no longer used by any node.
2022-09-28 15:06:13.179334610 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_333'. It is no longer used by any node.
2022-09-28 15:06:13.179350033 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1551'. It is no longer used by any node.
2022-09-28 15:06:13.179355771 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_668'. It is no longer used by any node.
2022-09-28 15:06:13.179359377 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_776'. It is no longer used by any node.
2022-09-28 15:06:13.179365551 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1091'. It is no longer used by any node.
2022-09-28 15:06:13.179370870 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_651'. It is no longer used by any node.
2022-09-28 15:06:13.179374735 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1548'. It is no longer used by any node.
2022-09-28 15:06:13.179378402 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1436'. It is no longer used by any node.
2022-09-28 15:06:13.179382049 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_868'. It is no longer used by any node.
2022-09-28 15:06:13.179385675 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_431'. It is no longer used by any node.
2022-09-28 15:06:13.179393633 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1433'. It is no longer used by any node.
2022-09-28 15:06:13.179399232 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1213'. It is no longer used by any node.
2022-09-28 15:06:13.179402772 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1531'. It is no longer used by any node.
2022-09-28 15:06:13.179406418 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1328'. It is no longer used by any node.
2022-09-28 15:06:13.179409739 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1311'. It is no longer used by any node.
2022-09-28 15:06:13.179786799 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.179794761 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.179799225 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.179803685 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.179808079 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.184173054 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.184192397 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.184198325 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.184203292 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.184208119 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.186280898 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.186289272 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.186293767 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.186298047 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.186302326 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.188315568 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.188322863 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.188329099 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.188333549 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.188337875 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.190334080 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.190342250 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.190347295 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.190351483 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.190361363 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.192366729 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.192375207 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.192380564 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.192384827 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.192388995 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.194459534 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.194473077 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.194480733 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.194485112 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.194489253 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.196508803 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.196521111 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.196527956 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.196532424 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.196536475 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.198622508 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.198633304 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.198638166 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.198642597 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.198647724 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.200750929 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.200767347 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.200772716 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.200778076 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.200783702 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:06:13.210639501 [V:onnxruntime:, session_state.cc:1186 VerifyEachNodeIsAssignedToAnEp] Node placements
2022-09-28 15:06:13.210688207 [V:onnxruntime:, session_state.cc:1193 VerifyEachNodeIsAssignedToAnEp]  Provider: [CPUExecutionProvider]: [Gather (Gather_173), Unsqueeze (Unsqueeze_174), Gather (Gather_179), Gather (Gather_185), Gather (Gather_188), Unsqueeze (Unsqueeze_191), Unsqueeze (Unsqueeze_193), Concat (Concat_194), Reshape (Reshape_196), Equal (Equal_201), Where (Where_202), Add (Add_211), Gather (Gather_219), Gather (Gather_222), Concat (Concat_230), Mul (Mul_242), Concat (Concat_245), Unsqueeze (Unsqueeze_251), Concat (Concat_252), Concat (Concat_263), Unsqueeze (Unsqueeze_266), Unsqueeze (Unsqueeze_267), Concat (Concat_268), Gather (Gather_286), Gather (Gather_289), Mul (Mul_295), Concat (Concat_298), Unsqueeze (Unsqueeze_304), Concat (Concat_305), Gather (Gather_312), Concat (Concat_318), Mul (Mul_322), Unsqueeze (Unsqueeze_323), Unsqueeze (Unsqueeze_325), Concat (Concat_326), Concat (Concat_332), Unsqueeze (Unsqueeze_335), Unsqueeze (Unsqueeze_336), Concat (Concat_337), Gather (Gather_373), Gather (Gather_376), Concat (Concat_384), Mul (Mul_396), Concat (Concat_399), Unsqueeze (Unsqueeze_405), Concat (Concat_406), Concat (Concat_417), Unsqueeze (Unsqueeze_420), Unsqueeze (Unsqueeze_421), Concat (Concat_422), Gather (Gather_440), Gather (Gather_443), Mul (Mul_449), Concat (Concat_452), Unsqueeze (Unsqueeze_458), Concat (Concat_459), Gather (Gather_466), Concat (Concat_472), Mul (Mul_476), Unsqueeze (Unsqueeze_477), Unsqueeze (Unsqueeze_479), Concat (Concat_480), Concat (Concat_486), Unsqueeze (Unsqueeze_489), Unsqueeze (Unsqueeze_490), Concat (Concat_491), Gather (Gather_527), Gather (Gather_530), Concat (Concat_538), Mul (Mul_550), Concat (Concat_553), Unsqueeze (Unsqueeze_559), Concat (Concat_560), Concat (Concat_571), Unsqueeze (Unsqueeze_574), Unsqueeze (Unsqueeze_575), Concat (Concat_576), Gather (Gather_594), Gather (Gather_597), Mul (Mul_603), Concat (Concat_606), Unsqueeze (Unsqueeze_612), Concat (Concat_613), Gather (Gather_620), Concat (Concat_626), Mul (Mul_630), Unsqueeze (Unsqueeze_631), Unsqueeze (Unsqueeze_633), Concat (Concat_634), Concat (Concat_640), Unsqueeze (Unsqueeze_643), Unsqueeze (Unsqueeze_644), Concat (Concat_645), Gather (Gather_681), Gather (Gather_684), Concat (Concat_692), Mul (Mul_704), Concat (Concat_707), Unsqueeze (Unsqueeze_713), Concat (Concat_714), Concat (Concat_725), Unsqueeze (Unsqueeze_728), Unsqueeze (Unsqueeze_729), Concat (Concat_730), Gather (Gather_748), Gather (Gather_751), Mul (Mul_757), Concat (Concat_760), Unsqueeze (Unsqueeze_766), Concat (Concat_767), Gather (Gather_774), Concat (Concat_780), Mul (Mul_784), Unsqueeze (Unsqueeze_785), Unsqueeze (Unsqueeze_787), Concat (Concat_788), Concat (Concat_794), Unsqueeze (Unsqueeze_797), Unsqueeze (Unsqueeze_798), Concat (Concat_799), Gather (Gather_835), Gather (Gather_838), Concat (Concat_846), Mul (Mul_858), Concat (Concat_861), Unsqueeze (Unsqueeze_867), Concat (Concat_868), Concat (Concat_879), Unsqueeze (Unsqueeze_882), Unsqueeze (Unsqueeze_883), Concat (Concat_884), Gather (Gather_902), Gather (Gather_905), Mul (Mul_911), Concat (Concat_914), Unsqueeze (Unsqueeze_920), Concat (Concat_921), Gather (Gather_928), Concat (Concat_934), Mul (Mul_938), Unsqueeze (Unsqueeze_939), Unsqueeze (Unsqueeze_941), Concat (Concat_942), Concat (Concat_948), Unsqueeze (Unsqueeze_951), Unsqueeze (Unsqueeze_952), Concat (Concat_953), Gather (Gather_989), Gather (Gather_992), Concat (Concat_1000), Mul (Mul_1012), Concat (Concat_1015), Unsqueeze (Unsqueeze_1021), Concat (Concat_1022), Concat (Concat_1033), Unsqueeze (Unsqueeze_1036), Unsqueeze (Unsqueeze_1037), Concat (Concat_1038), Gather (Gather_1056), Gather (Gather_1059), Mul (Mul_1065), Concat (Concat_1068), Unsqueeze (Unsqueeze_1074), Concat (Concat_1075), Gather (Gather_1082), Concat (Concat_1088), Mul (Mul_1092), Unsqueeze (Unsqueeze_1093), Unsqueeze (Unsqueeze_1095), Concat (Concat_1096), Concat (Concat_1102), Unsqueeze (Unsqueeze_1105), Unsqueeze (Unsqueeze_1106), Concat (Concat_1107), ]
2022-09-28 15:06:13.210721564 [V:onnxruntime:, session_state.cc:1193 VerifyEachNodeIsAssignedToAnEp]  Provider: [CUDAExecutionProvider]: [Shape (Shape_171), Reshape (Reshape_176), Shape (Shape_177), Gather (Gather_180), Mul (Mul_182), Shape (Shape_186), Unsqueeze (Unsqueeze_189), Unsqueeze (Unsqueeze_190), Expand (Expand_203), Cast (Cast_204), Sub (Sub_206), Cast (Cast_207), Where (Where_210), Range (Range_214), Gather (Gather_215), Add (Add_216), Shape (Shape_220), MatMul (MatMul_223), Add (Add_224), Mul (Mul_226), MatMul (MatMul_227), Add (Add_228), Reshape (Reshape_231), Transpose (Transpose_232), MatMul (MatMul_233), Add (Add_234), Reshape (Reshape_237), Transpose (Transpose_238), Concat (Concat_239), Concat (Concat_240), Reshape (Reshape_246), Transpose (Transpose_247), Reshape (Reshape_250), Reshape (Reshape_253), Reshape (Reshape_256), Softmax (Softmax_259), MatMul (MatMul_260), Reshape (Reshape_264), Transpose (Transpose_265), Reshape (Reshape_269), MatMul (MatMul_270), Add (Add_271), Add (Add_272), Shape (Shape_287), MatMul (MatMul_290), Add (Add_291), Mul (Mul_293), Reshape (Reshape_299), Transpose (Transpose_300), Reshape (Reshape_303), Reshape (Reshape_306), Reshape (Reshape_309), Shape (Shape_310), Reshape (Reshape_319), Add (Add_320), Reshape (Reshape_327), Softmax (Softmax_328), MatMul (MatMul_329), Reshape (Reshape_333), Transpose (Transpose_334), Reshape (Reshape_338), MatMul (MatMul_339), Add (Add_340), Add (Add_341), MatMul (MatMul_353), Add (Add_354), Sigmoid (Sigmoid_355), Mul (Mul_356), MatMul (MatMul_357), Add (Add_358), Add (Add_359), Shape (Shape_374), MatMul (MatMul_377), Add (Add_378), Mul (Mul_380), MatMul (MatMul_381), Add (Add_382), Reshape (Reshape_385), Transpose (Transpose_386), MatMul (MatMul_387), Add (Add_388), Reshape (Reshape_391), Transpose (Transpose_392), Concat (Concat_393), Concat (Concat_394), Reshape (Reshape_400), Transpose (Transpose_401), Reshape (Reshape_404), Reshape (Reshape_407), Reshape (Reshape_410), Softmax (Softmax_413), MatMul (MatMul_414), Reshape (Reshape_418), Transpose (Transpose_419), Reshape (Reshape_423), MatMul (MatMul_424), Add (Add_425), Add (Add_426), Shape (Shape_441), MatMul (MatMul_444), Add (Add_445), Mul (Mul_447), Reshape (Reshape_453), Transpose (Transpose_454), Reshape (Reshape_457), Reshape (Reshape_460), Reshape (Reshape_463), Shape (Shape_464), Reshape (Reshape_473), Add (Add_474), Reshape (Reshape_481), Softmax (Softmax_482), MatMul (MatMul_483), Reshape (Reshape_487), Transpose (Transpose_488), Reshape (Reshape_492), MatMul (MatMul_493), Add (Add_494), Add (Add_495), MatMul (MatMul_507), Add (Add_508), Sigmoid (Sigmoid_509), Mul (Mul_510), MatMul (MatMul_511), Add (Add_512), Add (Add_513), Shape (Shape_528), MatMul (MatMul_531), Add (Add_532), Mul (Mul_534), MatMul (MatMul_535), Add (Add_536), Reshape (Reshape_539), Transpose (Transpose_540), MatMul (MatMul_541), Add (Add_542), Reshape (Reshape_545), Transpose (Transpose_546), Concat (Concat_547), Concat (Concat_548), Reshape (Reshape_554), Transpose (Transpose_555), Reshape (Reshape_558), Reshape (Reshape_561), Reshape (Reshape_564), Softmax (Softmax_567), MatMul (MatMul_568), Reshape (Reshape_572), Transpose (Transpose_573), Reshape (Reshape_577), MatMul (MatMul_578), Add (Add_579), Add (Add_580), Shape (Shape_595), MatMul (MatMul_598), Add (Add_599), Mul (Mul_601), Reshape (Reshape_607), Transpose (Transpose_608), Reshape (Reshape_611), Reshape (Reshape_614), Reshape (Reshape_617), Shape (Shape_618), Reshape (Reshape_627), Add (Add_628), Reshape (Reshape_635), Softmax (Softmax_636), MatMul (MatMul_637), Reshape (Reshape_641), Transpose (Transpose_642), Reshape (Reshape_646), MatMul (MatMul_647), Add (Add_648), Add (Add_649), MatMul (MatMul_661), Add (Add_662), Sigmoid (Sigmoid_663), Mul (Mul_664), MatMul (MatMul_665), Add (Add_666), Add (Add_667), Shape (Shape_682), MatMul (MatMul_685), Add (Add_686), Mul (Mul_688), MatMul (MatMul_689), Add (Add_690), Reshape (Reshape_693), Transpose (Transpose_694), MatMul (MatMul_695), Add (Add_696), Reshape (Reshape_699), Transpose (Transpose_700), Concat (Concat_701), Concat (Concat_702), Reshape (Reshape_708), Transpose (Transpose_709), Reshape (Reshape_712), Reshape (Reshape_715), Reshape (Reshape_718), Softmax (Softmax_721), MatMul (MatMul_722), Reshape (Reshape_726), Transpose (Transpose_727), Reshape (Reshape_731), MatMul (MatMul_732), Add (Add_733), Add (Add_734), Shape (Shape_749), MatMul (MatMul_752), Add (Add_753), Mul (Mul_755), Reshape (Reshape_761), Transpose (Transpose_762), Reshape (Reshape_765), Reshape (Reshape_768), Reshape (Reshape_771), Shape (Shape_772), Reshape (Reshape_781), Add (Add_782), Reshape (Reshape_789), Softmax (Softmax_790), MatMul (MatMul_791), Reshape (Reshape_795), Transpose (Transpose_796), Reshape (Reshape_800), MatMul (MatMul_801), Add (Add_802), Add (Add_803), MatMul (MatMul_815), Add (Add_816), Sigmoid (Sigmoid_817), Mul (Mul_818), MatMul (MatMul_819), Add (Add_820), Add (Add_821), Shape (Shape_836), MatMul (MatMul_839), Add (Add_840), Mul (Mul_842), MatMul (MatMul_843), Add (Add_844), Reshape (Reshape_847), Transpose (Transpose_848), MatMul (MatMul_849), Add (Add_850), Reshape (Reshape_853), Transpose (Transpose_854), Concat (Concat_855), Concat (Concat_856), Reshape (Reshape_862), Transpose (Transpose_863), Reshape (Reshape_866), Reshape (Reshape_869), Reshape (Reshape_872), Softmax (Softmax_875), MatMul (MatMul_876), Reshape (Reshape_880), Transpose (Transpose_881), Reshape (Reshape_885), MatMul (MatMul_886), Add (Add_887), Add (Add_888), Shape (Shape_903), MatMul (MatMul_906), Add (Add_907), Mul (Mul_909), Reshape (Reshape_915), Transpose (Transpose_916), Reshape (Reshape_919), Reshape (Reshape_922), Reshape (Reshape_925), Shape (Shape_926), Reshape (Reshape_935), Add (Add_936), Reshape (Reshape_943), Softmax (Softmax_944), MatMul (MatMul_945), Reshape (Reshape_949), Transpose (Transpose_950), Reshape (Reshape_954), MatMul (MatMul_955), Add (Add_956), Add (Add_957), MatMul (MatMul_969), Add (Add_970), Sigmoid (Sigmoid_971), Mul (Mul_972), MatMul (MatMul_973), Add (Add_974), Add (Add_975), Shape (Shape_990), MatMul (MatMul_993), Add (Add_994), Mul (Mul_996), MatMul (MatMul_997), Add (Add_998), Reshape (Reshape_1001), Transpose (Transpose_1002), MatMul (MatMul_1003), Add (Add_1004), Reshape (Reshape_1007), Transpose (Transpose_1008), Concat (Concat_1009), Concat (Concat_1010), Reshape (Reshape_1016), Transpose (Transpose_1017), Reshape (Reshape_1020), Reshape (Reshape_1023), Reshape (Reshape_1026), Softmax (Softmax_1029), MatMul (MatMul_1030), Reshape (Reshape_1034), Transpose (Transpose_1035), Reshape (Reshape_1039), MatMul (MatMul_1040), Add (Add_1041), Add (Add_1042), Shape (Shape_1057), MatMul (MatMul_1060), Add (Add_1061), Mul (Mul_1063), Reshape (Reshape_1069), Transpose (Transpose_1070), Reshape (Reshape_1073), Reshape (Reshape_1076), Reshape (Reshape_1079), Shape (Shape_1080), Reshape (Reshape_1089), Add (Add_1090), Reshape (Reshape_1097), Softmax (Softmax_1098), MatMul (MatMul_1099), Reshape (Reshape_1103), Transpose (Transpose_1104), Reshape (Reshape_1108), MatMul (MatMul_1109), Add (Add_1110), Add (Add_1111), MatMul (MatMul_1123), Add (Add_1124), Sigmoid (Sigmoid_1125), Mul (Mul_1126), MatMul (MatMul_1127), Add (Add_1128), Add (Add_1129), MatMul (MatMul_1141), Add (Add_1143), Identity (Identity_1144), Identity (Identity_1145), Identity (Identity_1146), Identity (Identity_1147), Identity (Identity_1148), Identity (Identity_1149), Identity (Identity_1150), Identity (Identity_1151), Identity (Identity_1152), Identity (Identity_1153), Identity (Identity_1154), Identity (Identity_1155), LayerNormalization (LayerNormalization), LayerNormalization (LayerNormalization_token_0), LayerNormalization (LayerNormalization_token_1), LayerNormalization (LayerNormalization_token_2), LayerNormalization (LayerNormalization_token_3), LayerNormalization (LayerNormalization_token_4), LayerNormalization (LayerNormalization_token_5), LayerNormalization (LayerNormalization_token_6), LayerNormalization (LayerNormalization_token_7), LayerNormalization (LayerNormalization_token_8), LayerNormalization (LayerNormalization_token_9), LayerNormalization (LayerNormalization_token_10), LayerNormalization (LayerNormalization_token_11), LayerNormalization (LayerNormalization_token_12), LayerNormalization (LayerNormalization_token_13), LayerNormalization (LayerNormalization_token_14), LayerNormalization (LayerNormalization_token_15), LayerNormalization (LayerNormalization_token_16), FusedMatMul (MatMul_With_Transpose), FusedMatMul (MatMul_With_Transpose_token_17), FusedMatMul (MatMul_With_Transpose_token_18), FusedMatMul (MatMul_With_Transpose_token_19), FusedMatMul (MatMul_With_Transpose_token_20), FusedMatMul (MatMul_With_Transpose_token_21), FusedMatMul (MatMul_With_Transpose_token_22), FusedMatMul (MatMul_With_Transpose_token_23), FusedMatMul (MatMul_With_Transpose_token_24), FusedMatMul (MatMul_With_Transpose_token_25), FusedMatMul (MatMul_With_Transpose_token_26), FusedMatMul (MatMul_With_Transpose_token_27), ]
2022-09-28 15:06:13.216955746 [V:onnxruntime:, session_state.cc:81 CreateGraphInfo] SaveMLValueNameIndexMapping
2022-09-28 15:06:13.217201008 [V:onnxruntime:, session_state.cc:127 CreateGraphInfo] Done saving OrtValue mappings.
2022-09-28 15:06:13.220407995 [I:onnxruntime:, session_state_utils.cc:140 SaveInitializedTensors] Saving initialized tensors.
2022-09-28 15:06:13.390588129 [I:onnxruntime:, session_state_utils.cc:268 SaveInitializedTensors] Done saving initialized tensors
2022-09-28 15:06:13.399065894 [I:onnxruntime:, inference_session.cc:1518 Initialize] Session successfully initialized.
['CUDAExecutionProvider', 'CPUExecutionProvider']
Matthieu-Tinycoaching commented 2 years ago

Hi @fxmarty I opened an issue on Optimum repo as well : https://github.com/huggingface/optimum/issues/404

I use onnxruntime-gpu==1.12.1 and followed Nvidia guides to install CUDA 11.6 and cuDNN 8.4.1.50. When I run the command nvidi-smi I get:

Wed Sep 28 16:08:30 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.39.01    Driver Version: 510.39.01    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:21:00.0  On |                  N/A |
|  0%   49C    P8    50W / 390W |   4938MiB / 24576MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2707      G   /usr/lib/xorg/Xorg                 24MiB |
|    0   N/A  N/A      3007      G   /usr/bin/gnome-shell               86MiB |
|    0   N/A  N/A      5236      G   ...AAAAAAAAA= --shared-files       22MiB |
|    0   N/A  N/A      6720      G   /usr/lib/xorg/Xorg                459MiB |
|    0   N/A  N/A      6866      G   /usr/bin/gnome-shell               82MiB |
|    0   N/A  N/A     40623      G   ...rbird/254/thunderbird-bin      176MiB |
|    0   N/A  N/A     67590      G   ...AAAAAAAAA= --shared-files       27MiB |
|    0   N/A  N/A     76688      C   ...mers-gpu-fresh/bin/python     3613MiB |
|    0   N/A  N/A    111338      G   ...302758918310166069,131072      300MiB |
|    0   N/A  N/A    112042      G   ...RendererForSitePerProcess       76MiB |
|    0   N/A  N/A    118832      G   ...veSuggestionsOnlyOnDemand       60MiB |
+-----------------------------------------------------------------------------+
Matthieu-Tinycoaching commented 2 years ago

Hi @fxmarty ,

I ran your code below:

import onnxruntime

from optimum.onnxruntime import ORTModelForSeq2SeqLM

from transformers import MarianTokenizer

tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-fr-en")

options = onnxruntime.SessionOptions()
options.log_severity_level = 0  # verbose, to see which execution provider is used

ort_model = ORTModelForSeq2SeqLM.from_pretrained(
    "Helsinki-NLP/opus-mt-fr-en",
    from_transformers=True,
    provider="CUDAExecutionProvider",
    session_options=options,
)

print(ort_model.providers)

And got the following printed message:

/home/matthieu/anaconda3/envs/haystack-gpu-fresh/lib/python3.8/site-packages/transformers/models/marian/modeling_marian.py:234: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/home/matthieu/anaconda3/envs/haystack-gpu-fresh/lib/python3.8/site-packages/transformers/models/marian/modeling_marian.py:241: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attention_mask.size() != (bsz, 1, tgt_len, src_len):
/home/matthieu/anaconda3/envs/haystack-gpu-fresh/lib/python3.8/site-packages/transformers/models/marian/modeling_marian.py:273: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
/home/matthieu/anaconda3/envs/haystack-gpu-fresh/lib/python3.8/site-packages/transformers/models/marian/modeling_marian.py:856: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if input_shape[-1] > 1:
/home/matthieu/anaconda3/envs/haystack-gpu-fresh/lib/python3.8/site-packages/transformers/models/marian/modeling_marian.py:84: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  mask = torch.full((tgt_len, tgt_len), torch.tensor(torch.finfo(dtype).min))
2022-09-28 15:41:49.132057617 [I:onnxruntime:, inference_session.cc:262 operator()] Flush-to-zero and denormal-as-zero are off
2022-09-28 15:41:49.132084940 [I:onnxruntime:, inference_session.cc:270 ConstructorCommon] Creating and using per session threadpools since use_per_session_threads_ is true
2022-09-28 15:41:49.132091783 [I:onnxruntime:, inference_session.cc:291 ConstructorCommon] Dynamic block base set to 0
2022-09-28 15:41:50.104089436 [I:onnxruntime:, inference_session.cc:1246 Initialize] Initializing session.
2022-09-28 15:41:50.104112160 [I:onnxruntime:, inference_session.cc:1283 Initialize] Adding default CPU execution provider.
2022-09-28 15:41:50.112545917 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::ConstantOfShape_135'. It is no longer used by any node.
2022-09-28 15:41:50.112562158 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Mul_137'. It is no longer used by any node.
2022-09-28 15:41:50.112805669 [I:onnxruntime:, reshape_fusion.cc:38 ApplyImpl] Fused reshape node: input
2022-09-28 15:41:50.112912345 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 1
2022-09-28 15:41:50.115635156 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 0
2022-09-28 15:41:50.121531469 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_989'. It is no longer used by any node.
2022-09-28 15:41:50.121541819 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_986'. It is no longer used by any node.
2022-09-28 15:41:50.121545456 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_969'. It is no longer used by any node.
2022-09-28 15:41:50.121549093 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_966'. It is no longer used by any node.
2022-09-28 15:41:50.121553040 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_848'. It is no longer used by any node.
2022-09-28 15:41:50.121560836 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_707'. It is no longer used by any node.
2022-09-28 15:41:50.121565024 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_422'. It is no longer used by any node.
2022-09-28 15:41:50.121572759 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_543'. It is no longer used by any node.
2022-09-28 15:41:50.121579221 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_704'. It is no longer used by any node.
2022-09-28 15:41:50.121582908 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_845'. It is no longer used by any node.
2022-09-28 15:41:50.121589100 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_566'. It is no longer used by any node.
2022-09-28 15:41:50.121592727 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_281'. It is no longer used by any node.
2022-09-28 15:41:50.121597927 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_261'. It is no longer used by any node.
2022-09-28 15:41:50.121603408 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_264'. It is no longer used by any node.
2022-09-28 15:41:50.121615942 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_284'. It is no longer used by any node.
2022-09-28 15:41:50.121621042 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_402'. It is no longer used by any node.
2022-09-28 15:41:50.121625981 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_405'. It is no longer used by any node.
2022-09-28 15:41:50.121630600 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_425'. It is no longer used by any node.
2022-09-28 15:41:50.121635620 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_546'. It is no longer used by any node.
2022-09-28 15:41:50.121640379 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_563'. It is no longer used by any node.
2022-09-28 15:41:50.121645669 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_684'. It is no longer used by any node.
2022-09-28 15:41:50.121650399 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_828'. It is no longer used by any node.
2022-09-28 15:41:50.121655088 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_687'. It is no longer used by any node.
2022-09-28 15:41:50.121659997 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_825'. It is no longer used by any node.
2022-09-28 15:41:50.121876435 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.121883369 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.121889951 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.121896364 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.121901925 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.124614125 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.124623824 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.124628543 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.124632901 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.124637550 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.126129513 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.126138080 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.126142709 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.126149021 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.126154782 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.127618169 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.127625183 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.127629892 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.127634210 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.127638539 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.129098309 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.129105433 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.129109942 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.129114310 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.129118598 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.130616102 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.130624789 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.130629418 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.130633887 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.130638395 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.132091423 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.132098687 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.132103276 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.132107865 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.132112143 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.133823310 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.133833891 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.133841596 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.133854310 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.133860302 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.135340933 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.135348447 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.135352956 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.135357435 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.135361873 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.136832966 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.136839989 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.136844468 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.136849207 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.136853636 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:50.142334576 [V:onnxruntime:, session_state.cc:1186 VerifyEachNodeIsAssignedToAnEp] Node placements
2022-09-28 15:41:50.142348763 [V:onnxruntime:, session_state.cc:1193 VerifyEachNodeIsAssignedToAnEp]  Provider: [CPUExecutionProvider]: [Gather (Gather_107), Gather (Gather_120), Gather (Gather_123), Unsqueeze (Unsqueeze_126), Unsqueeze (Unsqueeze_128), Concat (Concat_129), Reshape (Reshape_131), Equal (Equal_136), Where (Where_137), Gather (Gather_148), Gather (Gather_151), Concat (Concat_159), Mul (Mul_169), Concat (Concat_172), Unsqueeze (Unsqueeze_178), Concat (Concat_179), Gather (Gather_186), Concat (Concat_192), Mul (Mul_196), Unsqueeze (Unsqueeze_197), Unsqueeze (Unsqueeze_199), Concat (Concat_200), Concat (Concat_206), Unsqueeze (Unsqueeze_209), Unsqueeze (Unsqueeze_210), Concat (Concat_211), Gather (Gather_247), Gather (Gather_250), Concat (Concat_258), Mul (Mul_268), Concat (Concat_271), Unsqueeze (Unsqueeze_277), Concat (Concat_278), Gather (Gather_285), Concat (Concat_291), Mul (Mul_295), Unsqueeze (Unsqueeze_296), Unsqueeze (Unsqueeze_298), Concat (Concat_299), Concat (Concat_305), Unsqueeze (Unsqueeze_308), Unsqueeze (Unsqueeze_309), Concat (Concat_310), Gather (Gather_346), Gather (Gather_349), Concat (Concat_357), Mul (Mul_367), Concat (Concat_370), Unsqueeze (Unsqueeze_376), Concat (Concat_377), Gather (Gather_384), Concat (Concat_390), Mul (Mul_394), Unsqueeze (Unsqueeze_395), Unsqueeze (Unsqueeze_397), Concat (Concat_398), Concat (Concat_404), Unsqueeze (Unsqueeze_407), Unsqueeze (Unsqueeze_408), Concat (Concat_409), Gather (Gather_445), Gather (Gather_448), Concat (Concat_456), Mul (Mul_466), Concat (Concat_469), Unsqueeze (Unsqueeze_475), Concat (Concat_476), Gather (Gather_483), Concat (Concat_489), Mul (Mul_493), Unsqueeze (Unsqueeze_494), Unsqueeze (Unsqueeze_496), Concat (Concat_497), Concat (Concat_503), Unsqueeze (Unsqueeze_506), Unsqueeze (Unsqueeze_507), Concat (Concat_508), Gather (Gather_544), Gather (Gather_547), Concat (Concat_555), Mul (Mul_565), Concat (Concat_568), Unsqueeze (Unsqueeze_574), Concat (Concat_575), Gather (Gather_582), Concat (Concat_588), Mul (Mul_592), Unsqueeze (Unsqueeze_593), Unsqueeze (Unsqueeze_595), Concat (Concat_596), Concat (Concat_602), Unsqueeze (Unsqueeze_605), Unsqueeze (Unsqueeze_606), Concat (Concat_607), Gather (Gather_643), Gather (Gather_646), Concat (Concat_654), Mul (Mul_664), Concat (Concat_667), Unsqueeze (Unsqueeze_673), Concat (Concat_674), Gather (Gather_681), Concat (Concat_687), Mul (Mul_691), Unsqueeze (Unsqueeze_692), Unsqueeze (Unsqueeze_694), Concat (Concat_695), Concat (Concat_701), Unsqueeze (Unsqueeze_704), Unsqueeze (Unsqueeze_705), Concat (Concat_706), ]
2022-09-28 15:41:50.142371287 [V:onnxruntime:, session_state.cc:1193 VerifyEachNodeIsAssignedToAnEp]  Provider: [CUDAExecutionProvider]: [Shape (Shape_105), Reshape (Reshape_110), Gather (Gather_111), Mul (Mul_113), Range (Range_115), Gather (Gather_116), Add (Add_117), Shape (Shape_121), Unsqueeze (Unsqueeze_124), Unsqueeze (Unsqueeze_125), Expand (Expand_138), Cast (Cast_139), Sub (Sub_141), Cast (Cast_142), Where (Where_145), Shape (Shape_149), MatMul (MatMul_152), Add (Add_153), Mul (Mul_155), MatMul (MatMul_156), Add (Add_157), Reshape (Reshape_160), Transpose (Transpose_161), MatMul (MatMul_162), Add (Add_163), Reshape (Reshape_166), Transpose (Transpose_167), Reshape (Reshape_173), Transpose (Transpose_174), Reshape (Reshape_177), Reshape (Reshape_180), Reshape (Reshape_183), Shape (Shape_184), Reshape (Reshape_193), Add (Add_194), Reshape (Reshape_201), Softmax (Softmax_202), MatMul (MatMul_203), Reshape (Reshape_207), Transpose (Transpose_208), Reshape (Reshape_212), MatMul (MatMul_213), Add (Add_214), Add (Add_215), MatMul (MatMul_227), Add (Add_228), Sigmoid (Sigmoid_229), Mul (Mul_230), MatMul (MatMul_231), Add (Add_232), Add (Add_233), Shape (Shape_248), MatMul (MatMul_251), Add (Add_252), Mul (Mul_254), MatMul (MatMul_255), Add (Add_256), Reshape (Reshape_259), Transpose (Transpose_260), MatMul (MatMul_261), Add (Add_262), Reshape (Reshape_265), Transpose (Transpose_266), Reshape (Reshape_272), Transpose (Transpose_273), Reshape (Reshape_276), Reshape (Reshape_279), Reshape (Reshape_282), Shape (Shape_283), Reshape (Reshape_292), Add (Add_293), Reshape (Reshape_300), Softmax (Softmax_301), MatMul (MatMul_302), Reshape (Reshape_306), Transpose (Transpose_307), Reshape (Reshape_311), MatMul (MatMul_312), Add (Add_313), Add (Add_314), MatMul (MatMul_326), Add (Add_327), Sigmoid (Sigmoid_328), Mul (Mul_329), MatMul (MatMul_330), Add (Add_331), Add (Add_332), Shape (Shape_347), MatMul (MatMul_350), Add (Add_351), Mul (Mul_353), MatMul (MatMul_354), Add (Add_355), Reshape (Reshape_358), Transpose (Transpose_359), MatMul (MatMul_360), Add (Add_361), Reshape (Reshape_364), Transpose (Transpose_365), Reshape (Reshape_371), Transpose (Transpose_372), Reshape (Reshape_375), Reshape (Reshape_378), Reshape (Reshape_381), Shape (Shape_382), Reshape (Reshape_391), Add (Add_392), Reshape (Reshape_399), Softmax (Softmax_400), MatMul (MatMul_401), Reshape (Reshape_405), Transpose (Transpose_406), Reshape (Reshape_410), MatMul (MatMul_411), Add (Add_412), Add (Add_413), MatMul (MatMul_425), Add (Add_426), Sigmoid (Sigmoid_427), Mul (Mul_428), MatMul (MatMul_429), Add (Add_430), Add (Add_431), Shape (Shape_446), MatMul (MatMul_449), Add (Add_450), Mul (Mul_452), MatMul (MatMul_453), Add (Add_454), Reshape (Reshape_457), Transpose (Transpose_458), MatMul (MatMul_459), Add (Add_460), Reshape (Reshape_463), Transpose (Transpose_464), Reshape (Reshape_470), Transpose (Transpose_471), Reshape (Reshape_474), Reshape (Reshape_477), Reshape (Reshape_480), Shape (Shape_481), Reshape (Reshape_490), Add (Add_491), Reshape (Reshape_498), Softmax (Softmax_499), MatMul (MatMul_500), Reshape (Reshape_504), Transpose (Transpose_505), Reshape (Reshape_509), MatMul (MatMul_510), Add (Add_511), Add (Add_512), MatMul (MatMul_524), Add (Add_525), Sigmoid (Sigmoid_526), Mul (Mul_527), MatMul (MatMul_528), Add (Add_529), Add (Add_530), Shape (Shape_545), MatMul (MatMul_548), Add (Add_549), Mul (Mul_551), MatMul (MatMul_552), Add (Add_553), Reshape (Reshape_556), Transpose (Transpose_557), MatMul (MatMul_558), Add (Add_559), Reshape (Reshape_562), Transpose (Transpose_563), Reshape (Reshape_569), Transpose (Transpose_570), Reshape (Reshape_573), Reshape (Reshape_576), Reshape (Reshape_579), Shape (Shape_580), Reshape (Reshape_589), Add (Add_590), Reshape (Reshape_597), Softmax (Softmax_598), MatMul (MatMul_599), Reshape (Reshape_603), Transpose (Transpose_604), Reshape (Reshape_608), MatMul (MatMul_609), Add (Add_610), Add (Add_611), MatMul (MatMul_623), Add (Add_624), Sigmoid (Sigmoid_625), Mul (Mul_626), MatMul (MatMul_627), Add (Add_628), Add (Add_629), Shape (Shape_644), MatMul (MatMul_647), Add (Add_648), Mul (Mul_650), MatMul (MatMul_651), Add (Add_652), Reshape (Reshape_655), Transpose (Transpose_656), MatMul (MatMul_657), Add (Add_658), Reshape (Reshape_661), Transpose (Transpose_662), Reshape (Reshape_668), Transpose (Transpose_669), Reshape (Reshape_672), Reshape (Reshape_675), Reshape (Reshape_678), Shape (Shape_679), Reshape (Reshape_688), Add (Add_689), Reshape (Reshape_696), Softmax (Softmax_697), MatMul (MatMul_698), Reshape (Reshape_702), Transpose (Transpose_703), Reshape (Reshape_707), MatMul (MatMul_708), Add (Add_709), Add (Add_710), MatMul (MatMul_722), Add (Add_723), Sigmoid (Sigmoid_724), Mul (Mul_725), MatMul (MatMul_726), Add (Add_727), Add (Add_728), LayerNormalization (LayerNormalization), LayerNormalization (LayerNormalization_token_0), LayerNormalization (LayerNormalization_token_1), LayerNormalization (LayerNormalization_token_2), LayerNormalization (LayerNormalization_token_3), LayerNormalization (LayerNormalization_token_4), LayerNormalization (LayerNormalization_token_5), LayerNormalization (LayerNormalization_token_6), LayerNormalization (LayerNormalization_token_7), LayerNormalization (LayerNormalization_token_8), LayerNormalization (LayerNormalization_token_9), LayerNormalization (LayerNormalization_token_10), FusedMatMul (MatMul_With_Transpose), FusedMatMul (MatMul_With_Transpose_token_11), FusedMatMul (MatMul_With_Transpose_token_12), FusedMatMul (MatMul_With_Transpose_token_13), FusedMatMul (MatMul_With_Transpose_token_14), FusedMatMul (MatMul_With_Transpose_token_15), ]
2022-09-28 15:41:50.143315581 [V:onnxruntime:, session_state.cc:81 CreateGraphInfo] SaveMLValueNameIndexMapping
2022-09-28 15:41:50.143395165 [V:onnxruntime:, session_state.cc:127 CreateGraphInfo] Done saving OrtValue mappings.
2022-09-28 15:41:51.425170275 [I:onnxruntime:, session_state_utils.cc:140 SaveInitializedTensors] Saving initialized tensors.
2022-09-28 15:41:51.490961150 [I:onnxruntime:, session_state_utils.cc:268 SaveInitializedTensors] Done saving initialized tensors
2022-09-28 15:41:51.496423045 [I:onnxruntime:, inference_session.cc:1518 Initialize] Session successfully initialized.
2022-09-28 15:41:51.496565510 [I:onnxruntime:, inference_session.cc:270 ConstructorCommon] Creating and using per session threadpools since use_per_session_threads_ is true
2022-09-28 15:41:51.496575830 [I:onnxruntime:, inference_session.cc:291 ConstructorCommon] Dynamic block base set to 0
2022-09-28 15:41:51.704116054 [I:onnxruntime:, inference_session.cc:1246 Initialize] Initializing session.
2022-09-28 15:41:51.704136393 [I:onnxruntime:, inference_session.cc:1283 Initialize] Adding default CPU execution provider.
2022-09-28 15:41:51.715012479 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::ConstantOfShape_217'. It is no longer used by any node.
2022-09-28 15:41:51.715028900 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::ConstantOfShape_241'. It is no longer used by any node.
2022-09-28 15:41:51.715038719 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Mul_219'. It is no longer used by any node.
2022-09-28 15:41:51.715045202 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Mul_243'. It is no longer used by any node.
2022-09-28 15:41:51.715546460 [I:onnxruntime:, reshape_fusion.cc:38 ApplyImpl] Fused reshape node: input
2022-09-28 15:41:51.715567781 [I:onnxruntime:, reshape_fusion.cc:38 ApplyImpl] Fused reshape node: onnx::Less_201
2022-09-28 15:41:51.715773529 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 2
2022-09-28 15:41:51.718255174 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Slice_192'. It is no longer used by any node.
2022-09-28 15:41:51.718271586 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Slice_193'. It is no longer used by any node.
2022-09-28 15:41:51.718278319 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Slice_194'. It is no longer used by any node.
2022-09-28 15:41:51.721045856 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 0
2022-09-28 15:41:51.730915758 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1688'. It is no longer used by any node.
2022-09-28 15:41:51.730925467 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1567'. It is no longer used by any node.
2022-09-28 15:41:51.730929134 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1547'. It is no longer used by any node.
2022-09-28 15:41:51.730935607 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1544'. It is no longer used by any node.
2022-09-28 15:41:51.730941839 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1426'. It is no longer used by any node.
2022-09-28 15:41:51.730947309 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1305'. It is no longer used by any node.
2022-09-28 15:41:51.730953040 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1302'. It is no longer used by any node.
2022-09-28 15:41:51.730958541 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1282'. It is no longer used by any node.
2022-09-28 15:41:51.730963701 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1806'. It is no longer used by any node.
2022-09-28 15:41:51.730967418 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1043'. It is no longer used by any node.
2022-09-28 15:41:51.730972528 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1023'. It is no longer used by any node.
2022-09-28 15:41:51.730976165 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1020'. It is no longer used by any node.
2022-09-28 15:41:51.730980052 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_899'. It is no longer used by any node.
2022-09-28 15:41:51.730985192 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_781'. It is no longer used by any node.
2022-09-28 15:41:51.730988920 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_778'. It is no longer used by any node.
2022-09-28 15:41:51.730993889 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_761'. It is no longer used by any node.
2022-09-28 15:41:51.730998508 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_519'. It is no longer used by any node.
2022-09-28 15:41:51.731003568 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_516'. It is no longer used by any node.
2022-09-28 15:41:51.731008167 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_499'. It is no longer used by any node.
2022-09-28 15:41:51.731012916 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_496'. It is no longer used by any node.
2022-09-28 15:41:51.731018126 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_378'. It is no longer used by any node.
2022-09-28 15:41:51.731022865 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_375'. It is no longer used by any node.
2022-09-28 15:41:51.731031251 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1161'. It is no longer used by any node.
2022-09-28 15:41:51.731036391 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_640'. It is no longer used by any node.
2022-09-28 15:41:51.731041681 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1164'. It is no longer used by any node.
2022-09-28 15:41:51.731046771 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_758'. It is no longer used by any node.
2022-09-28 15:41:51.731051931 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1685'. It is no longer used by any node.
2022-09-28 15:41:51.731057662 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_637'. It is no longer used by any node.
2022-09-28 15:41:51.731062532 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1829'. It is no longer used by any node.
2022-09-28 15:41:51.731068243 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1285'. It is no longer used by any node.
2022-09-28 15:41:51.731073102 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1809'. It is no longer used by any node.
2022-09-28 15:41:51.731079424 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_902'. It is no longer used by any node.
2022-09-28 15:41:51.731084164 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1040'. It is no longer used by any node.
2022-09-28 15:41:51.731089794 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1423'. It is no longer used by any node.
2022-09-28 15:41:51.731096197 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1564'. It is no longer used by any node.
2022-09-28 15:41:51.731100996 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1826'. It is no longer used by any node.
2022-09-28 15:41:51.731504235 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.731511539 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.731517541 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.731524815 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.731530857 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.736761263 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.736771523 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.736776372 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.736786992 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.736793395 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.739694792 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.739702506 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.739707376 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.739713478 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.739726383 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.742635995 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.742643700 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.742648399 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.742654912 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.742661996 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.745559224 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.745566548 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.745572940 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.745581066 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.745596105 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.748498393 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.748506318 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.748511017 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.748515666 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.748526577 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.751436349 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.751443994 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.751448894 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.751455196 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.751461839 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.754364908 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.754372542 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.754377372 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.754383694 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.754390647 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.757295470 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.757303035 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.757309617 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.757315789 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.757323414 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.760251473 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.760262985 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.760269878 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.760276140 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.760288043 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:51.774117445 [V:onnxruntime:, session_state.cc:1186 VerifyEachNodeIsAssignedToAnEp] Node placements
2022-09-28 15:41:51.774144447 [V:onnxruntime:, session_state.cc:1193 VerifyEachNodeIsAssignedToAnEp]  Provider: [CUDAExecutionProvider]: [Shape (Shape_218), Reshape (Reshape_223), Gather (Gather_224), Mul (Mul_226), ConstantOfShape (ConstantOfShape_230), Range (Range_240), Add (Add_242), Shape (Shape_243), Reshape (Reshape_251), Less (Less_252), Where (Where_255), Unsqueeze (Unsqueeze_257), Unsqueeze (Unsqueeze_258), Expand (Expand_271), Shape (Shape_276), Unsqueeze (Unsqueeze_279), Unsqueeze (Unsqueeze_280), Expand (Expand_293), Cast (Cast_294), Sub (Sub_296), Cast (Cast_297), Where (Where_300), Range (Range_302), Gather (Gather_303), Add (Add_304), Shape (Shape_308), MatMul (MatMul_311), Add (Add_312), Mul (Mul_314), MatMul (MatMul_315), Add (Add_316), Reshape (Reshape_319), Transpose (Transpose_320), MatMul (MatMul_321), Add (Add_322), Reshape (Reshape_325), Transpose (Transpose_326), Reshape (Reshape_332), Transpose (Transpose_333), Reshape (Reshape_336), Reshape (Reshape_339), Reshape (Reshape_342), Shape (Shape_343), Reshape (Reshape_352), Add (Add_353), Reshape (Reshape_360), Softmax (Softmax_361), MatMul (MatMul_362), Reshape (Reshape_366), Transpose (Transpose_367), Reshape (Reshape_371), MatMul (MatMul_372), Add (Add_373), Add (Add_374), Shape (Shape_389), MatMul (MatMul_392), Add (Add_393), Mul (Mul_395), MatMul (MatMul_396), Add (Add_397), Reshape (Reshape_400), Transpose (Transpose_401), MatMul (MatMul_402), Add (Add_403), Reshape (Reshape_406), Transpose (Transpose_407), Reshape (Reshape_413), Transpose (Transpose_414), Reshape (Reshape_417), Reshape (Reshape_420), Reshape (Reshape_423), Shape (Shape_424), Reshape (Reshape_433), Add (Add_434), Reshape (Reshape_441), Softmax (Softmax_442), MatMul (MatMul_443), Reshape (Reshape_447), Transpose (Transpose_448), Reshape (Reshape_452), MatMul (MatMul_453), Add (Add_454), Add (Add_455), MatMul (MatMul_467), Add (Add_468), Sigmoid (Sigmoid_469), Mul (Mul_470), MatMul (MatMul_471), Add (Add_472), Add (Add_473), Shape (Shape_488), MatMul (MatMul_491), Add (Add_492), Mul (Mul_494), MatMul (MatMul_495), Add (Add_496), Reshape (Reshape_499), Transpose (Transpose_500), MatMul (MatMul_501), Add (Add_502), Reshape (Reshape_505), Transpose (Transpose_506), Reshape (Reshape_512), Transpose (Transpose_513), Reshape (Reshape_516), Reshape (Reshape_519), Reshape (Reshape_522), Shape (Shape_523), Reshape (Reshape_532), Add (Add_533), Reshape (Reshape_540), Softmax (Softmax_541), MatMul (MatMul_542), Reshape (Reshape_546), Transpose (Transpose_547), Reshape (Reshape_551), MatMul (MatMul_552), Add (Add_553), Add (Add_554), Shape (Shape_569), MatMul (MatMul_572), Add (Add_573), Mul (Mul_575), MatMul (MatMul_576), Add (Add_577), Reshape (Reshape_580), Transpose (Transpose_581), MatMul (MatMul_582), Add (Add_583), Reshape (Reshape_586), Transpose (Transpose_587), Reshape (Reshape_593), Transpose (Transpose_594), Reshape (Reshape_597), Reshape (Reshape_600), Reshape (Reshape_603), Shape (Shape_604), Reshape (Reshape_613), Add (Add_614), Reshape (Reshape_621), Softmax (Softmax_622), MatMul (MatMul_623), Reshape (Reshape_627), Transpose (Transpose_628), Reshape (Reshape_632), MatMul (MatMul_633), Add (Add_634), Add (Add_635), MatMul (MatMul_647), Add (Add_648), Sigmoid (Sigmoid_649), Mul (Mul_650), MatMul (MatMul_651), Add (Add_652), Add (Add_653), Shape (Shape_668), MatMul (MatMul_671), Add (Add_672), Mul (Mul_674), MatMul (MatMul_675), Add (Add_676), Reshape (Reshape_679), Transpose (Transpose_680), MatMul (MatMul_681), Add (Add_682), Reshape (Reshape_685), Transpose (Transpose_686), Reshape (Reshape_692), Transpose (Transpose_693), Reshape (Reshape_696), Reshape (Reshape_699), Reshape (Reshape_702), Shape (Shape_703), Reshape (Reshape_712), Add (Add_713), Reshape (Reshape_720), Softmax (Softmax_721), MatMul (MatMul_722), Reshape (Reshape_726), Transpose (Transpose_727), Reshape (Reshape_731), MatMul (MatMul_732), Add (Add_733), Add (Add_734), Shape (Shape_749), MatMul (MatMul_752), Add (Add_753), Mul (Mul_755), MatMul (MatMul_756), Add (Add_757), Reshape (Reshape_760), Transpose (Transpose_761), MatMul (MatMul_762), Add (Add_763), Reshape (Reshape_766), Transpose (Transpose_767), Reshape (Reshape_773), Transpose (Transpose_774), Reshape (Reshape_777), Reshape (Reshape_780), Reshape (Reshape_783), Shape (Shape_784), Reshape (Reshape_793), Add (Add_794), Reshape (Reshape_801), Softmax (Softmax_802), MatMul (MatMul_803), Reshape (Reshape_807), Transpose (Transpose_808), Reshape (Reshape_812), MatMul (MatMul_813), Add (Add_814), Add (Add_815), MatMul (MatMul_827), Add (Add_828), Sigmoid (Sigmoid_829), Mul (Mul_830), MatMul (MatMul_831), Add (Add_832), Add (Add_833), Shape (Shape_848), MatMul (MatMul_851), Add (Add_852), Mul (Mul_854), MatMul (MatMul_855), Add (Add_856), Reshape (Reshape_859), Transpose (Transpose_860), MatMul (MatMul_861), Add (Add_862), Reshape (Reshape_865), Transpose (Transpose_866), Reshape (Reshape_872), Transpose (Transpose_873), Reshape (Reshape_876), Reshape (Reshape_879), Reshape (Reshape_882), Shape (Shape_883), Reshape (Reshape_892), Add (Add_893), Reshape (Reshape_900), Softmax (Softmax_901), MatMul (MatMul_902), Reshape (Reshape_906), Transpose (Transpose_907), Reshape (Reshape_911), MatMul (MatMul_912), Add (Add_913), Add (Add_914), Shape (Shape_929), MatMul (MatMul_932), Add (Add_933), Mul (Mul_935), MatMul (MatMul_936), Add (Add_937), Reshape (Reshape_940), Transpose (Transpose_941), MatMul (MatMul_942), Add (Add_943), Reshape (Reshape_946), Transpose (Transpose_947), Reshape (Reshape_953), Transpose (Transpose_954), Reshape (Reshape_957), Reshape (Reshape_960), Reshape (Reshape_963), Shape (Shape_964), Reshape (Reshape_973), Add (Add_974), Reshape (Reshape_981), Softmax (Softmax_982), MatMul (MatMul_983), Reshape (Reshape_987), Transpose (Transpose_988), Reshape (Reshape_992), MatMul (MatMul_993), Add (Add_994), Add (Add_995), MatMul (MatMul_1007), Add (Add_1008), Sigmoid (Sigmoid_1009), Mul (Mul_1010), MatMul (MatMul_1011), Add (Add_1012), Add (Add_1013), Shape (Shape_1028), MatMul (MatMul_1031), Add (Add_1032), Mul (Mul_1034), MatMul (MatMul_1035), Add (Add_1036), Reshape (Reshape_1039), Transpose (Transpose_1040), MatMul (MatMul_1041), Add (Add_1042), Reshape (Reshape_1045), Transpose (Transpose_1046), Reshape (Reshape_1052), Transpose (Transpose_1053), Reshape (Reshape_1056), Reshape (Reshape_1059), Reshape (Reshape_1062), Shape (Shape_1063), Reshape (Reshape_1072), Add (Add_1073), Reshape (Reshape_1080), Softmax (Softmax_1081), MatMul (MatMul_1082), Reshape (Reshape_1086), Transpose (Transpose_1087), Reshape (Reshape_1091), MatMul (MatMul_1092), Add (Add_1093), Add (Add_1094), Shape (Shape_1109), MatMul (MatMul_1112), Add (Add_1113), Mul (Mul_1115), MatMul (MatMul_1116), Add (Add_1117), Reshape (Reshape_1120), Transpose (Transpose_1121), MatMul (MatMul_1122), Add (Add_1123), Reshape (Reshape_1126), Transpose (Transpose_1127), Reshape (Reshape_1133), Transpose (Transpose_1134), Reshape (Reshape_1137), Reshape (Reshape_1140), Reshape (Reshape_1143), Shape (Shape_1144), Reshape (Reshape_1153), Add (Add_1154), Reshape (Reshape_1161), Softmax (Softmax_1162), MatMul (MatMul_1163), Reshape (Reshape_1167), Transpose (Transpose_1168), Reshape (Reshape_1172), MatMul (MatMul_1173), Add (Add_1174), Add (Add_1175), MatMul (MatMul_1187), Add (Add_1188), Sigmoid (Sigmoid_1189), Mul (Mul_1190), MatMul (MatMul_1191), Add (Add_1192), Add (Add_1193), Shape (Shape_1208), MatMul (MatMul_1211), Add (Add_1212), Mul (Mul_1214), MatMul (MatMul_1215), Add (Add_1216), Reshape (Reshape_1219), Transpose (Transpose_1220), MatMul (MatMul_1221), Add (Add_1222), Reshape (Reshape_1225), Transpose (Transpose_1226), Reshape (Reshape_1232), Transpose (Transpose_1233), Reshape (Reshape_1236), Reshape (Reshape_1239), Reshape (Reshape_1242), Shape (Shape_1243), Reshape (Reshape_1252), Add (Add_1253), Reshape (Reshape_1260), Softmax (Softmax_1261), MatMul (MatMul_1262), Reshape (Reshape_1266), Transpose (Transpose_1267), Reshape (Reshape_1271), MatMul (MatMul_1272), Add (Add_1273), Add (Add_1274), Shape (Shape_1289), MatMul (MatMul_1292), Add (Add_1293), Mul (Mul_1295), MatMul (MatMul_1296), Add (Add_1297), Reshape (Reshape_1300), Transpose (Transpose_1301), MatMul (MatMul_1302), Add (Add_1303), Reshape (Reshape_1306), Transpose (Transpose_1307), Reshape (Reshape_1313), Transpose (Transpose_1314), Reshape (Reshape_1317), Reshape (Reshape_1320), Reshape (Reshape_1323), Shape (Shape_1324), Reshape (Reshape_1333), Add (Add_1334), Reshape (Reshape_1341), Softmax (Softmax_1342), MatMul (MatMul_1343), Reshape (Reshape_1347), Transpose (Transpose_1348), Reshape (Reshape_1352), MatMul (MatMul_1353), Add (Add_1354), Add (Add_1355), MatMul (MatMul_1367), Add (Add_1368), Sigmoid (Sigmoid_1369), Mul (Mul_1370), MatMul (MatMul_1371), Add (Add_1372), Add (Add_1373), MatMul (MatMul_1385), Add (Add_1387), LayerNormalization (LayerNormalization), LayerNormalization (LayerNormalization_token_0), LayerNormalization (LayerNormalization_token_1), LayerNormalization (LayerNormalization_token_2), LayerNormalization (LayerNormalization_token_3), LayerNormalization (LayerNormalization_token_4), LayerNormalization (LayerNormalization_token_5), LayerNormalization (LayerNormalization_token_6), LayerNormalization (LayerNormalization_token_7), LayerNormalization (LayerNormalization_token_8), LayerNormalization (LayerNormalization_token_9), LayerNormalization (LayerNormalization_token_10), LayerNormalization (LayerNormalization_token_11), LayerNormalization (LayerNormalization_token_12), LayerNormalization (LayerNormalization_token_13), LayerNormalization (LayerNormalization_token_14), LayerNormalization (LayerNormalization_token_15), LayerNormalization (LayerNormalization_token_16), FusedMatMul (MatMul_With_Transpose), FusedMatMul (MatMul_With_Transpose_token_17), FusedMatMul (MatMul_With_Transpose_token_18), FusedMatMul (MatMul_With_Transpose_token_19), FusedMatMul (MatMul_With_Transpose_token_20), FusedMatMul (MatMul_With_Transpose_token_21), FusedMatMul (MatMul_With_Transpose_token_22), FusedMatMul (MatMul_With_Transpose_token_23), FusedMatMul (MatMul_With_Transpose_token_24), FusedMatMul (MatMul_With_Transpose_token_25), FusedMatMul (MatMul_With_Transpose_token_26), FusedMatMul (MatMul_With_Transpose_token_27), ]
2022-09-28 15:41:51.774227167 [V:onnxruntime:, session_state.cc:1193 VerifyEachNodeIsAssignedToAnEp]  Provider: [CPUExecutionProvider]: [Gather (Gather_217), Gather (Gather_220), Unsqueeze (Unsqueeze_221), Concat (Concat_229), Slice (Slice_235), Squeeze (Squeeze_236), Unsqueeze (Unsqueeze_259), Concat (Concat_262), Reshape (Reshape_264), Equal (Equal_269), Where (Where_270), Gather (Gather_275), Gather (Gather_278), Unsqueeze (Unsqueeze_281), Unsqueeze (Unsqueeze_283), Concat (Concat_284), Reshape (Reshape_286), Equal (Equal_291), Where (Where_292), Gather (Gather_307), Gather (Gather_310), Concat (Concat_318), Mul (Mul_328), Concat (Concat_331), Unsqueeze (Unsqueeze_337), Concat (Concat_338), Gather (Gather_345), Concat (Concat_351), Mul (Mul_355), Unsqueeze (Unsqueeze_356), Unsqueeze (Unsqueeze_358), Concat (Concat_359), Concat (Concat_365), Unsqueeze (Unsqueeze_368), Unsqueeze (Unsqueeze_369), Concat (Concat_370), Gather (Gather_388), Gather (Gather_391), Concat (Concat_399), Mul (Mul_409), Concat (Concat_412), Unsqueeze (Unsqueeze_418), Concat (Concat_419), Gather (Gather_426), Concat (Concat_432), Mul (Mul_436), Unsqueeze (Unsqueeze_437), Unsqueeze (Unsqueeze_439), Concat (Concat_440), Concat (Concat_446), Unsqueeze (Unsqueeze_449), Unsqueeze (Unsqueeze_450), Concat (Concat_451), Gather (Gather_487), Gather (Gather_490), Concat (Concat_498), Mul (Mul_508), Concat (Concat_511), Unsqueeze (Unsqueeze_517), Concat (Concat_518), Gather (Gather_525), Concat (Concat_531), Mul (Mul_535), Unsqueeze (Unsqueeze_536), Unsqueeze (Unsqueeze_538), Concat (Concat_539), Concat (Concat_545), Unsqueeze (Unsqueeze_548), Unsqueeze (Unsqueeze_549), Concat (Concat_550), Gather (Gather_568), Gather (Gather_571), Concat (Concat_579), Mul (Mul_589), Concat (Concat_592), Unsqueeze (Unsqueeze_598), Concat (Concat_599), Gather (Gather_606), Concat (Concat_612), Mul (Mul_616), Unsqueeze (Unsqueeze_617), Unsqueeze (Unsqueeze_619), Concat (Concat_620), Concat (Concat_626), Unsqueeze (Unsqueeze_629), Unsqueeze (Unsqueeze_630), Concat (Concat_631), Gather (Gather_667), Gather (Gather_670), Concat (Concat_678), Mul (Mul_688), Concat (Concat_691), Unsqueeze (Unsqueeze_697), Concat (Concat_698), Gather (Gather_705), Concat (Concat_711), Mul (Mul_715), Unsqueeze (Unsqueeze_716), Unsqueeze (Unsqueeze_718), Concat (Concat_719), Concat (Concat_725), Unsqueeze (Unsqueeze_728), Unsqueeze (Unsqueeze_729), Concat (Concat_730), Gather (Gather_748), Gather (Gather_751), Concat (Concat_759), Mul (Mul_769), Concat (Concat_772), Unsqueeze (Unsqueeze_778), Concat (Concat_779), Gather (Gather_786), Concat (Concat_792), Mul (Mul_796), Unsqueeze (Unsqueeze_797), Unsqueeze (Unsqueeze_799), Concat (Concat_800), Concat (Concat_806), Unsqueeze (Unsqueeze_809), Unsqueeze (Unsqueeze_810), Concat (Concat_811), Gather (Gather_847), Gather (Gather_850), Concat (Concat_858), Mul (Mul_868), Concat (Concat_871), Unsqueeze (Unsqueeze_877), Concat (Concat_878), Gather (Gather_885), Concat (Concat_891), Mul (Mul_895), Unsqueeze (Unsqueeze_896), Unsqueeze (Unsqueeze_898), Concat (Concat_899), Concat (Concat_905), Unsqueeze (Unsqueeze_908), Unsqueeze (Unsqueeze_909), Concat (Concat_910), Gather (Gather_928), Gather (Gather_931), Concat (Concat_939), Mul (Mul_949), Concat (Concat_952), Unsqueeze (Unsqueeze_958), Concat (Concat_959), Gather (Gather_966), Concat (Concat_972), Mul (Mul_976), Unsqueeze (Unsqueeze_977), Unsqueeze (Unsqueeze_979), Concat (Concat_980), Concat (Concat_986), Unsqueeze (Unsqueeze_989), Unsqueeze (Unsqueeze_990), Concat (Concat_991), Gather (Gather_1027), Gather (Gather_1030), Concat (Concat_1038), Mul (Mul_1048), Concat (Concat_1051), Unsqueeze (Unsqueeze_1057), Concat (Concat_1058), Gather (Gather_1065), Concat (Concat_1071), Mul (Mul_1075), Unsqueeze (Unsqueeze_1076), Unsqueeze (Unsqueeze_1078), Concat (Concat_1079), Concat (Concat_1085), Unsqueeze (Unsqueeze_1088), Unsqueeze (Unsqueeze_1089), Concat (Concat_1090), Gather (Gather_1108), Gather (Gather_1111), Concat (Concat_1119), Mul (Mul_1129), Concat (Concat_1132), Unsqueeze (Unsqueeze_1138), Concat (Concat_1139), Gather (Gather_1146), Concat (Concat_1152), Mul (Mul_1156), Unsqueeze (Unsqueeze_1157), Unsqueeze (Unsqueeze_1159), Concat (Concat_1160), Concat (Concat_1166), Unsqueeze (Unsqueeze_1169), Unsqueeze (Unsqueeze_1170), Concat (Concat_1171), Gather (Gather_1207), Gather (Gather_1210), Concat (Concat_1218), Mul (Mul_1228), Concat (Concat_1231), Unsqueeze (Unsqueeze_1237), Concat (Concat_1238), Gather (Gather_1245), Concat (Concat_1251), Mul (Mul_1255), Unsqueeze (Unsqueeze_1256), Unsqueeze (Unsqueeze_1258), Concat (Concat_1259), Concat (Concat_1265), Unsqueeze (Unsqueeze_1268), Unsqueeze (Unsqueeze_1269), Concat (Concat_1270), Gather (Gather_1288), Gather (Gather_1291), Concat (Concat_1299), Mul (Mul_1309), Concat (Concat_1312), Unsqueeze (Unsqueeze_1318), Concat (Concat_1319), Gather (Gather_1326), Concat (Concat_1332), Mul (Mul_1336), Unsqueeze (Unsqueeze_1337), Unsqueeze (Unsqueeze_1339), Concat (Concat_1340), Concat (Concat_1346), Unsqueeze (Unsqueeze_1349), Unsqueeze (Unsqueeze_1350), Concat (Concat_1351), ]
2022-09-28 15:41:51.776110887 [V:onnxruntime:, session_state.cc:81 CreateGraphInfo] SaveMLValueNameIndexMapping
2022-09-28 15:41:51.776256679 [V:onnxruntime:, session_state.cc:127 CreateGraphInfo] Done saving OrtValue mappings.
2022-09-28 15:41:51.778236626 [I:onnxruntime:, session_state_utils.cc:140 SaveInitializedTensors] Saving initialized tensors.
2022-09-28 15:41:51.871922612 [I:onnxruntime:, session_state_utils.cc:268 SaveInitializedTensors] Done saving initialized tensors
2022-09-28 15:41:51.880265423 [I:onnxruntime:, inference_session.cc:1518 Initialize] Session successfully initialized.
2022-09-28 15:41:51.880428047 [I:onnxruntime:, inference_session.cc:270 ConstructorCommon] Creating and using per session threadpools since use_per_session_threads_ is true
2022-09-28 15:41:51.880438647 [I:onnxruntime:, inference_session.cc:291 ConstructorCommon] Dynamic block base set to 0
2022-09-28 15:41:52.065442615 [I:onnxruntime:, inference_session.cc:1246 Initialize] Initializing session.
2022-09-28 15:41:52.065462354 [I:onnxruntime:, inference_session.cc:1283 Initialize] Adding default CPU execution provider.
2022-09-28 15:41:52.074337131 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::ConstantOfShape_215'. It is no longer used by any node.
2022-09-28 15:41:52.074354986 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Mul_217'. It is no longer used by any node.
2022-09-28 15:41:52.074810977 [I:onnxruntime:, reshape_fusion.cc:38 ApplyImpl] Fused reshape node: input
2022-09-28 15:41:52.074990354 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 1
2022-09-28 15:41:52.079535547 [I:onnxruntime:, reshape_fusion.cc:42 ApplyImpl] Total fused reshape node count: 0
2022-09-28 15:41:52.088051151 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1548'. It is no longer used by any node.
2022-09-28 15:41:52.088061431 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1531'. It is no longer used by any node.
2022-09-28 15:41:52.088065208 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1528'. It is no longer used by any node.
2022-09-28 15:41:52.088069426 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1331'. It is no longer used by any node.
2022-09-28 15:41:52.088082081 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1311'. It is no longer used by any node.
2022-09-28 15:41:52.088091108 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1308'. It is no longer used by any node.
2022-09-28 15:41:52.088096258 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1091'. It is no longer used by any node.
2022-09-28 15:41:52.088101218 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1088'. It is no longer used by any node.
2022-09-28 15:41:52.088104915 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_871'. It is no longer used by any node.
2022-09-28 15:41:52.088109974 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1108'. It is no longer used by any node.
2022-09-28 15:41:52.088113692 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1551'. It is no longer used by any node.
2022-09-28 15:41:52.088118852 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_776'. It is no longer used by any node.
2022-09-28 15:41:52.088122408 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1436'. It is no longer used by any node.
2022-09-28 15:41:52.088127408 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_671'. It is no longer used by any node.
2022-09-28 15:41:52.088130975 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_668'. It is no longer used by any node.
2022-09-28 15:41:52.088136035 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_556'. It is no longer used by any node.
2022-09-28 15:41:52.088140734 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_868'. It is no longer used by any node.
2022-09-28 15:41:52.088146966 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_888'. It is no longer used by any node.
2022-09-28 15:41:52.088153018 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_773'. It is no longer used by any node.
2022-09-28 15:41:52.088158328 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_993'. It is no longer used by any node.
2022-09-28 15:41:52.088163478 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1111'. It is no longer used by any node.
2022-09-28 15:41:52.088170592 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1328'. It is no longer used by any node.
2022-09-28 15:41:52.088174369 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1433'. It is no longer used by any node.
2022-09-28 15:41:52.088178417 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_651'. It is no longer used by any node.
2022-09-28 15:41:52.088183727 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_996'. It is no longer used by any node.
2022-09-28 15:41:52.088192835 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_431'. It is no longer used by any node.
2022-09-28 15:41:52.088197554 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_1213'. It is no longer used by any node.
2022-09-28 15:41:52.088202854 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_648'. It is no longer used by any node.
2022-09-28 15:41:52.088206351 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_553'. It is no longer used by any node.
2022-09-28 15:41:52.088213725 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_428'. It is no longer used by any node.
2022-09-28 15:41:52.088219065 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_1216'. It is no longer used by any node.
2022-09-28 15:41:52.088224035 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_891'. It is no longer used by any node.
2022-09-28 15:41:52.088229445 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_333'. It is no longer used by any node.
2022-09-28 15:41:52.088234465 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_336'. It is no longer used by any node.
2022-09-28 15:41:52.088238202 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Pow_448'. It is no longer used by any node.
2022-09-28 15:41:52.088242821 [I:onnxruntime:, graph.cc:3497 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Add_451'. It is no longer used by any node.
2022-09-28 15:41:52.088575604 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.088582888 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.088589090 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.088595222 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.088601695 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.092879201 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.092889180 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.092895612 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.092901895 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.092907806 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.095295730 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.095306541 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.095311621 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.095318585 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.095324626 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.097664698 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.097672192 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.097676991 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.097683404 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.097692822 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.100043484 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.100051349 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.100056198 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.100062150 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.100069855 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.102412672 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.102420787 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.102425667 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.102431809 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.102439654 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.104769585 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.104777160 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.104782140 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.104787981 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.104794023 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.107136389 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.107144134 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.107150426 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.107156518 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.107167740 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.109487503 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.109494917 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.109501480 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.109507211 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.109513022 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.111849246 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.111857051 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.111863033 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.111869185 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.111912439 [V:onnxruntime:, attention_fusion.cc:631 FuseSubGraph] Faild to find path v
2022-09-28 15:41:52.122360837 [V:onnxruntime:, session_state.cc:1186 VerifyEachNodeIsAssignedToAnEp] Node placements
2022-09-28 15:41:52.122378130 [V:onnxruntime:, session_state.cc:1193 VerifyEachNodeIsAssignedToAnEp]  Provider: [CPUExecutionProvider]: [Gather (Gather_173), Unsqueeze (Unsqueeze_174), Gather (Gather_179), Gather (Gather_185), Gather (Gather_188), Unsqueeze (Unsqueeze_191), Unsqueeze (Unsqueeze_193), Concat (Concat_194), Reshape (Reshape_196), Equal (Equal_201), Where (Where_202), Add (Add_211), Gather (Gather_219), Gather (Gather_222), Concat (Concat_230), Mul (Mul_242), Concat (Concat_245), Unsqueeze (Unsqueeze_251), Concat (Concat_252), Concat (Concat_263), Unsqueeze (Unsqueeze_266), Unsqueeze (Unsqueeze_267), Concat (Concat_268), Gather (Gather_286), Gather (Gather_289), Mul (Mul_295), Concat (Concat_298), Unsqueeze (Unsqueeze_304), Concat (Concat_305), Gather (Gather_312), Concat (Concat_318), Mul (Mul_322), Unsqueeze (Unsqueeze_323), Unsqueeze (Unsqueeze_325), Concat (Concat_326), Concat (Concat_332), Unsqueeze (Unsqueeze_335), Unsqueeze (Unsqueeze_336), Concat (Concat_337), Gather (Gather_373), Gather (Gather_376), Concat (Concat_384), Mul (Mul_396), Concat (Concat_399), Unsqueeze (Unsqueeze_405), Concat (Concat_406), Concat (Concat_417), Unsqueeze (Unsqueeze_420), Unsqueeze (Unsqueeze_421), Concat (Concat_422), Gather (Gather_440), Gather (Gather_443), Mul (Mul_449), Concat (Concat_452), Unsqueeze (Unsqueeze_458), Concat (Concat_459), Gather (Gather_466), Concat (Concat_472), Mul (Mul_476), Unsqueeze (Unsqueeze_477), Unsqueeze (Unsqueeze_479), Concat (Concat_480), Concat (Concat_486), Unsqueeze (Unsqueeze_489), Unsqueeze (Unsqueeze_490), Concat (Concat_491), Gather (Gather_527), Gather (Gather_530), Concat (Concat_538), Mul (Mul_550), Concat (Concat_553), Unsqueeze (Unsqueeze_559), Concat (Concat_560), Concat (Concat_571), Unsqueeze (Unsqueeze_574), Unsqueeze (Unsqueeze_575), Concat (Concat_576), Gather (Gather_594), Gather (Gather_597), Mul (Mul_603), Concat (Concat_606), Unsqueeze (Unsqueeze_612), Concat (Concat_613), Gather (Gather_620), Concat (Concat_626), Mul (Mul_630), Unsqueeze (Unsqueeze_631), Unsqueeze (Unsqueeze_633), Concat (Concat_634), Concat (Concat_640), Unsqueeze (Unsqueeze_643), Unsqueeze (Unsqueeze_644), Concat (Concat_645), Gather (Gather_681), Gather (Gather_684), Concat (Concat_692), Mul (Mul_704), Concat (Concat_707), Unsqueeze (Unsqueeze_713), Concat (Concat_714), Concat (Concat_725), Unsqueeze (Unsqueeze_728), Unsqueeze (Unsqueeze_729), Concat (Concat_730), Gather (Gather_748), Gather (Gather_751), Mul (Mul_757), Concat (Concat_760), Unsqueeze (Unsqueeze_766), Concat (Concat_767), Gather (Gather_774), Concat (Concat_780), Mul (Mul_784), Unsqueeze (Unsqueeze_785), Unsqueeze (Unsqueeze_787), Concat (Concat_788), Concat (Concat_794), Unsqueeze (Unsqueeze_797), Unsqueeze (Unsqueeze_798), Concat (Concat_799), Gather (Gather_835), Gather (Gather_838), Concat (Concat_846), Mul (Mul_858), Concat (Concat_861), Unsqueeze (Unsqueeze_867), Concat (Concat_868), Concat (Concat_879), Unsqueeze (Unsqueeze_882), Unsqueeze (Unsqueeze_883), Concat (Concat_884), Gather (Gather_902), Gather (Gather_905), Mul (Mul_911), Concat (Concat_914), Unsqueeze (Unsqueeze_920), Concat (Concat_921), Gather (Gather_928), Concat (Concat_934), Mul (Mul_938), Unsqueeze (Unsqueeze_939), Unsqueeze (Unsqueeze_941), Concat (Concat_942), Concat (Concat_948), Unsqueeze (Unsqueeze_951), Unsqueeze (Unsqueeze_952), Concat (Concat_953), Gather (Gather_989), Gather (Gather_992), Concat (Concat_1000), Mul (Mul_1012), Concat (Concat_1015), Unsqueeze (Unsqueeze_1021), Concat (Concat_1022), Concat (Concat_1033), Unsqueeze (Unsqueeze_1036), Unsqueeze (Unsqueeze_1037), Concat (Concat_1038), Gather (Gather_1056), Gather (Gather_1059), Mul (Mul_1065), Concat (Concat_1068), Unsqueeze (Unsqueeze_1074), Concat (Concat_1075), Gather (Gather_1082), Concat (Concat_1088), Mul (Mul_1092), Unsqueeze (Unsqueeze_1093), Unsqueeze (Unsqueeze_1095), Concat (Concat_1096), Concat (Concat_1102), Unsqueeze (Unsqueeze_1105), Unsqueeze (Unsqueeze_1106), Concat (Concat_1107), ]
2022-09-28 15:41:52.122409130 [V:onnxruntime:, session_state.cc:1193 VerifyEachNodeIsAssignedToAnEp]  Provider: [CUDAExecutionProvider]: [Shape (Shape_171), Reshape (Reshape_176), Shape (Shape_177), Gather (Gather_180), Mul (Mul_182), Shape (Shape_186), Unsqueeze (Unsqueeze_189), Unsqueeze (Unsqueeze_190), Expand (Expand_203), Cast (Cast_204), Sub (Sub_206), Cast (Cast_207), Where (Where_210), Range (Range_214), Gather (Gather_215), Add (Add_216), Shape (Shape_220), MatMul (MatMul_223), Add (Add_224), Mul (Mul_226), MatMul (MatMul_227), Add (Add_228), Reshape (Reshape_231), Transpose (Transpose_232), MatMul (MatMul_233), Add (Add_234), Reshape (Reshape_237), Transpose (Transpose_238), Concat (Concat_239), Concat (Concat_240), Reshape (Reshape_246), Transpose (Transpose_247), Reshape (Reshape_250), Reshape (Reshape_253), Reshape (Reshape_256), Softmax (Softmax_259), MatMul (MatMul_260), Reshape (Reshape_264), Transpose (Transpose_265), Reshape (Reshape_269), MatMul (MatMul_270), Add (Add_271), Add (Add_272), Shape (Shape_287), MatMul (MatMul_290), Add (Add_291), Mul (Mul_293), Reshape (Reshape_299), Transpose (Transpose_300), Reshape (Reshape_303), Reshape (Reshape_306), Reshape (Reshape_309), Shape (Shape_310), Reshape (Reshape_319), Add (Add_320), Reshape (Reshape_327), Softmax (Softmax_328), MatMul (MatMul_329), Reshape (Reshape_333), Transpose (Transpose_334), Reshape (Reshape_338), MatMul (MatMul_339), Add (Add_340), Add (Add_341), MatMul (MatMul_353), Add (Add_354), Sigmoid (Sigmoid_355), Mul (Mul_356), MatMul (MatMul_357), Add (Add_358), Add (Add_359), Shape (Shape_374), MatMul (MatMul_377), Add (Add_378), Mul (Mul_380), MatMul (MatMul_381), Add (Add_382), Reshape (Reshape_385), Transpose (Transpose_386), MatMul (MatMul_387), Add (Add_388), Reshape (Reshape_391), Transpose (Transpose_392), Concat (Concat_393), Concat (Concat_394), Reshape (Reshape_400), Transpose (Transpose_401), Reshape (Reshape_404), Reshape (Reshape_407), Reshape (Reshape_410), Softmax (Softmax_413), MatMul (MatMul_414), Reshape (Reshape_418), Transpose (Transpose_419), Reshape (Reshape_423), MatMul (MatMul_424), Add (Add_425), Add (Add_426), Shape (Shape_441), MatMul (MatMul_444), Add (Add_445), Mul (Mul_447), Reshape (Reshape_453), Transpose (Transpose_454), Reshape (Reshape_457), Reshape (Reshape_460), Reshape (Reshape_463), Shape (Shape_464), Reshape (Reshape_473), Add (Add_474), Reshape (Reshape_481), Softmax (Softmax_482), MatMul (MatMul_483), Reshape (Reshape_487), Transpose (Transpose_488), Reshape (Reshape_492), MatMul (MatMul_493), Add (Add_494), Add (Add_495), MatMul (MatMul_507), Add (Add_508), Sigmoid (Sigmoid_509), Mul (Mul_510), MatMul (MatMul_511), Add (Add_512), Add (Add_513), Shape (Shape_528), MatMul (MatMul_531), Add (Add_532), Mul (Mul_534), MatMul (MatMul_535), Add (Add_536), Reshape (Reshape_539), Transpose (Transpose_540), MatMul (MatMul_541), Add (Add_542), Reshape (Reshape_545), Transpose (Transpose_546), Concat (Concat_547), Concat (Concat_548), Reshape (Reshape_554), Transpose (Transpose_555), Reshape (Reshape_558), Reshape (Reshape_561), Reshape (Reshape_564), Softmax (Softmax_567), MatMul (MatMul_568), Reshape (Reshape_572), Transpose (Transpose_573), Reshape (Reshape_577), MatMul (MatMul_578), Add (Add_579), Add (Add_580), Shape (Shape_595), MatMul (MatMul_598), Add (Add_599), Mul (Mul_601), Reshape (Reshape_607), Transpose (Transpose_608), Reshape (Reshape_611), Reshape (Reshape_614), Reshape (Reshape_617), Shape (Shape_618), Reshape (Reshape_627), Add (Add_628), Reshape (Reshape_635), Softmax (Softmax_636), MatMul (MatMul_637), Reshape (Reshape_641), Transpose (Transpose_642), Reshape (Reshape_646), MatMul (MatMul_647), Add (Add_648), Add (Add_649), MatMul (MatMul_661), Add (Add_662), Sigmoid (Sigmoid_663), Mul (Mul_664), MatMul (MatMul_665), Add (Add_666), Add (Add_667), Shape (Shape_682), MatMul (MatMul_685), Add (Add_686), Mul (Mul_688), MatMul (MatMul_689), Add (Add_690), Reshape (Reshape_693), Transpose (Transpose_694), MatMul (MatMul_695), Add (Add_696), Reshape (Reshape_699), Transpose (Transpose_700), Concat (Concat_701), Concat (Concat_702), Reshape (Reshape_708), Transpose (Transpose_709), Reshape (Reshape_712), Reshape (Reshape_715), Reshape (Reshape_718), Softmax (Softmax_721), MatMul (MatMul_722), Reshape (Reshape_726), Transpose (Transpose_727), Reshape (Reshape_731), MatMul (MatMul_732), Add (Add_733), Add (Add_734), Shape (Shape_749), MatMul (MatMul_752), Add (Add_753), Mul (Mul_755), Reshape (Reshape_761), Transpose (Transpose_762), Reshape (Reshape_765), Reshape (Reshape_768), Reshape (Reshape_771), Shape (Shape_772), Reshape (Reshape_781), Add (Add_782), Reshape (Reshape_789), Softmax (Softmax_790), MatMul (MatMul_791), Reshape (Reshape_795), Transpose (Transpose_796), Reshape (Reshape_800), MatMul (MatMul_801), Add (Add_802), Add (Add_803), MatMul (MatMul_815), Add (Add_816), Sigmoid (Sigmoid_817), Mul (Mul_818), MatMul (MatMul_819), Add (Add_820), Add (Add_821), Shape (Shape_836), MatMul (MatMul_839), Add (Add_840), Mul (Mul_842), MatMul (MatMul_843), Add (Add_844), Reshape (Reshape_847), Transpose (Transpose_848), MatMul (MatMul_849), Add (Add_850), Reshape (Reshape_853), Transpose (Transpose_854), Concat (Concat_855), Concat (Concat_856), Reshape (Reshape_862), Transpose (Transpose_863), Reshape (Reshape_866), Reshape (Reshape_869), Reshape (Reshape_872), Softmax (Softmax_875), MatMul (MatMul_876), Reshape (Reshape_880), Transpose (Transpose_881), Reshape (Reshape_885), MatMul (MatMul_886), Add (Add_887), Add (Add_888), Shape (Shape_903), MatMul (MatMul_906), Add (Add_907), Mul (Mul_909), Reshape (Reshape_915), Transpose (Transpose_916), Reshape (Reshape_919), Reshape (Reshape_922), Reshape (Reshape_925), Shape (Shape_926), Reshape (Reshape_935), Add (Add_936), Reshape (Reshape_943), Softmax (Softmax_944), MatMul (MatMul_945), Reshape (Reshape_949), Transpose (Transpose_950), Reshape (Reshape_954), MatMul (MatMul_955), Add (Add_956), Add (Add_957), MatMul (MatMul_969), Add (Add_970), Sigmoid (Sigmoid_971), Mul (Mul_972), MatMul (MatMul_973), Add (Add_974), Add (Add_975), Shape (Shape_990), MatMul (MatMul_993), Add (Add_994), Mul (Mul_996), MatMul (MatMul_997), Add (Add_998), Reshape (Reshape_1001), Transpose (Transpose_1002), MatMul (MatMul_1003), Add (Add_1004), Reshape (Reshape_1007), Transpose (Transpose_1008), Concat (Concat_1009), Concat (Concat_1010), Reshape (Reshape_1016), Transpose (Transpose_1017), Reshape (Reshape_1020), Reshape (Reshape_1023), Reshape (Reshape_1026), Softmax (Softmax_1029), MatMul (MatMul_1030), Reshape (Reshape_1034), Transpose (Transpose_1035), Reshape (Reshape_1039), MatMul (MatMul_1040), Add (Add_1041), Add (Add_1042), Shape (Shape_1057), MatMul (MatMul_1060), Add (Add_1061), Mul (Mul_1063), Reshape (Reshape_1069), Transpose (Transpose_1070), Reshape (Reshape_1073), Reshape (Reshape_1076), Reshape (Reshape_1079), Shape (Shape_1080), Reshape (Reshape_1089), Add (Add_1090), Reshape (Reshape_1097), Softmax (Softmax_1098), MatMul (MatMul_1099), Reshape (Reshape_1103), Transpose (Transpose_1104), Reshape (Reshape_1108), MatMul (MatMul_1109), Add (Add_1110), Add (Add_1111), MatMul (MatMul_1123), Add (Add_1124), Sigmoid (Sigmoid_1125), Mul (Mul_1126), MatMul (MatMul_1127), Add (Add_1128), Add (Add_1129), MatMul (MatMul_1141), Add (Add_1143), Identity (Identity_1144), Identity (Identity_1145), Identity (Identity_1146), Identity (Identity_1147), Identity (Identity_1148), Identity (Identity_1149), Identity (Identity_1150), Identity (Identity_1151), Identity (Identity_1152), Identity (Identity_1153), Identity (Identity_1154), Identity (Identity_1155), LayerNormalization (LayerNormalization), LayerNormalization (LayerNormalization_token_0), LayerNormalization (LayerNormalization_token_1), LayerNormalization (LayerNormalization_token_2), LayerNormalization (LayerNormalization_token_3), LayerNormalization (LayerNormalization_token_4), LayerNormalization (LayerNormalization_token_5), LayerNormalization (LayerNormalization_token_6), LayerNormalization (LayerNormalization_token_7), LayerNormalization (LayerNormalization_token_8), LayerNormalization (LayerNormalization_token_9), LayerNormalization (LayerNormalization_token_10), LayerNormalization (LayerNormalization_token_11), LayerNormalization (LayerNormalization_token_12), LayerNormalization (LayerNormalization_token_13), LayerNormalization (LayerNormalization_token_14), LayerNormalization (LayerNormalization_token_15), LayerNormalization (LayerNormalization_token_16), FusedMatMul (MatMul_With_Transpose), FusedMatMul (MatMul_With_Transpose_token_17), FusedMatMul (MatMul_With_Transpose_token_18), FusedMatMul (MatMul_With_Transpose_token_19), FusedMatMul (MatMul_With_Transpose_token_20), FusedMatMul (MatMul_With_Transpose_token_21), FusedMatMul (MatMul_With_Transpose_token_22), FusedMatMul (MatMul_With_Transpose_token_23), FusedMatMul (MatMul_With_Transpose_token_24), FusedMatMul (MatMul_With_Transpose_token_25), FusedMatMul (MatMul_With_Transpose_token_26), FusedMatMul (MatMul_With_Transpose_token_27), ]
2022-09-28 15:41:52.123805909 [V:onnxruntime:, session_state.cc:81 CreateGraphInfo] SaveMLValueNameIndexMapping
2022-09-28 15:41:52.123912244 [V:onnxruntime:, session_state.cc:127 CreateGraphInfo] Done saving OrtValue mappings.
2022-09-28 15:41:52.125376443 [I:onnxruntime:, session_state_utils.cc:140 SaveInitializedTensors] Saving initialized tensors.
2022-09-28 15:41:52.211905251 [I:onnxruntime:, session_state_utils.cc:268 SaveInitializedTensors] Done saving initialized tensors
2022-09-28 15:41:52.219352322 [I:onnxruntime:, inference_session.cc:1518 Initialize] Session successfully initialized.
['CUDAExecutionProvider', 'CPUExecutionProvider']

So, it seems that GPU is working also?

Is it important to provide provider="CUDAExecutionProvider" when exporting the torch model to ONNX in?:

ort_model = ORTModelForSeq2SeqLM.from_pretrained(
    "Helsinki-NLP/opus-mt-fr-en",
    from_transformers=True,
    provider="CUDAExecutionProvider",
    session_options=options,
)
fxmarty commented 2 years ago

@Matthieu-Tinycoaching Let's move the discussion to https://github.com/huggingface/optimum/issues/404