Issue while exporting internlm2-chat-7b model to OpenVINO format

[lumurillo]

When trying to export the internlm2-chat-7b model to the OpenVINO format, got an error.

The environment used was the following:

• OS: Windows 11
• Python: 3.11
• OpenVINO: 2024.3.0.dev20240705
• optimum: 1.21.1
• optimum-intel: 1.19.0.dev0+eeb1df0

The error log is the following:

optimum-cli export openvino --weight-format int4 --sym --trust-remote-code --convert-tokenizer --model internlm/internlm2-chat-7b internlm2-chat-7b-int4 --convert-tokenizer option is deprecated. Tokenizer will be converted by default.
Framework not specified. Using pt to export the model.
model-00001-of-00008.safetensors: 100%|███████████████████████████████████████| 1.95G/1.95G [01:47<00:00, 14.3MB/s]
model-00002-of-00008.safetensors: 100%|███████████████████████████████████████| 1.95G/1.95G [01:32<00:00, 20.9MB/s]
model-00003-of-00008.safetensors: 100%|███████████████████████████████████████| 1.98G/1.98G [01:56<00:00, 17.0MB/s]
model-00004-of-00008.safetensors: 100%|███████████████████████████████████████| 1.95G/1.95G [02:25<00:00, 13.4MB/s]
model-00005-of-00008.safetensors: 100%|███████████████████████████████████████| 1.98G/1.98G [01:53<00:00, 17.4MB/s]
model-00006-of-00008.safetensors: 100%|███████████████████████████████████████| 1.95G/1.95G [01:51<00:00, 17.4MB/s]
model-00007-of-00008.safetensors: 100%|███████████████████████████████████████| 1.98G/1.98G [01:52<00:00, 17.5MB/s]
model-00008-of-00008.safetensors: 100%|███████████████████████████████████████| 1.75G/1.75G [01:32<00:00, 19.0MB/s]
Downloading shards: 100%|███████████████████████████████████████████████████████████| 8/8 [14:59<00:00, 112.39s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████| 8/8 [00:22<00:00,  2.75s/it]
generation_config.json: 100%|█████████████████████████████████████████████████████████████| 123/123 [00:00<?, ?B/s]
Automatic task detection to text-generation-with-past (possible synonyms are: causal-lm-with-past).
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████| 2.51k/2.51k [00:00<?, ?B/s]
tokenization_internlm2_fast.py: 100%|█████████████████████████████████████████████████| 7.80k/7.80k [00:00<?, ?B/s]
tokenization_internlm2.py: 100%|███████████████████████████████████████████████| 8.81k/8.81k [00:00<00:00, 893kB/s]
A new version of the following files was downloaded from https://huggingface.co/internlm/internlm2-chat-7b:
- tokenization_internlm2.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
A new version of the following files was downloaded from https://huggingface.co/internlm/internlm2-chat-7b:
- tokenization_internlm2_fast.py
- tokenization_internlm2.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
tokenizer.model: 100%|████████████████████████████████████████████████████████| 1.48M/1.48M [00:00<00:00, 3.20MB/s]
special_tokens_map.json: 100%|████████████████████████████████████████████████████████████| 713/713 [00:00<?, ?B/s]
Using framework PyTorch: 2.3.1+cpu
Overriding 1 configuration item(s)
        - use_cache -> True
C:\Users\Local_Admin\.cache\huggingface\modules\transformers_modules\internlm\internlm2-chat-7b\70e6cdc9643ce7e3d9a369fb984dc5f1a1b2cec6\modeling_internlm2.py:1095: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if sequence_length != 1:
Export model to OpenVINO directly failed with:
Couldn't get TorchScript module by tracing. With exception:
'tuple' object has no attribute 'shape'
Please check correctness of provided 'example_input'.  You can also provide TorchScript module that you obtained yourself, please refer to PyTorch documentation: https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html..
Model will be exported to ONNX
[ WARNING ] Making stateful models is not supported when exporting to ONNX as an intermediate step. A stateless model will be exported instead. It may result in sub-optimal inference performance.Provide a model that can be converted to OpenVINO without fallback to ONNX conversion path.
Using framework PyTorch: 2.3.1+cpu
Overriding 1 configuration item(s)
        - use_cache -> True
Traceback (most recent call last):
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\openvino\frontend\pytorch\ts_decoder.py", line 41, in __init__
    pt_module = self._get_scripted_model(
                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\openvino\frontend\pytorch\ts_decoder.py", line 134, in _get_scripted_model
    scripted = torch.jit.trace(
               ^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\jit\_trace.py", line 820, in trace
    return trace_module(
           ^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\jit\_trace.py", line 1088, in trace_module
    module._c._create_method_from_trace(
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\nncf\torch\dynamic_graph\wrappers.py", line 146, in wrapped
    return module_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1522, in _slow_forward
    result = self.forward(*input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\exporters\openvino\convert.py", line 354, in ts_patched_forward
    outputs = patched_forward(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\exporters\onnx\model_patcher.py", line 152, in patched_forward
    outputs = self.orig_forward(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\.cache\huggingface\modules\transformers_modules\internlm\internlm2-chat-7b\70e6cdc9643ce7e3d9a369fb984dc5f1a1b2cec6\modeling_internlm2.py", line 1204, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\nncf\torch\dynamic_graph\wrappers.py", line 146, in wrapped
    return module_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1522, in _slow_forward
    result = self.forward(*input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\.cache\huggingface\modules\transformers_modules\internlm\internlm2-chat-7b\70e6cdc9643ce7e3d9a369fb984dc5f1a1b2cec6\modeling_internlm2.py", line 1004, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\nncf\torch\dynamic_graph\wrappers.py", line 146, in wrapped
    return module_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1522, in _slow_forward
    result = self.forward(*input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\.cache\huggingface\modules\transformers_modules\internlm\internlm2-chat-7b\70e6cdc9643ce7e3d9a369fb984dc5f1a1b2cec6\modeling_internlm2.py", line 738, in forward
    hidden_states, self_attn_weights, present_key_value = self.attention(
                                                          ^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\nncf\torch\dynamic_graph\wrappers.py", line 146, in wrapped
    return module_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1522, in _slow_forward
    result = self.forward(*input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\exporters\openvino\model_patcher.py", line 1117, in _internlm2_attention_forward
    kv_seq_len += past_key_value[0].shape[-2]
                  ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'tuple' object has no attribute 'shape'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\exporters\openvino\convert.py", line 367, in export_pytorch
    ov_model = convert_model(model, example_input=dummy_inputs, input=input_info)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\openvino\tools\ovc\convert.py", line 100, in convert_model
    ov_model, _ = _convert(cli_parser, params, True)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\openvino\tools\ovc\convert_impl.py", line 548, in _convert
    raise e
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\openvino\tools\ovc\convert_impl.py", line 457, in _convert
    get_pytorch_decoder(args['input_model'], example_inputs, args)
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\openvino\tools\ovc\moc_frontend\pytorch_frontend_utils.py", line 54, in get_pytorch_decoder
    decoder = TorchScriptPythonDecoder(
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\openvino\frontend\pytorch\ts_decoder.py", line 52, in __init__
    raise RuntimeError(
RuntimeError: Couldn't get TorchScript module by tracing. With exception:
'tuple' object has no attribute 'shape'
Please check correctness of provided 'example_input'.  You can also provide TorchScript module that you obtained yourself, please refer to PyTorch documentation: https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Scripts\optimum-cli.exe\__main__.py", line 7, in <module>
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\commands\optimum_cli.py", line 208, in main
    service.run()
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\commands\export\openvino.py", line 362, in run
    main_export(
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\exporters\openvino\__main__.py", line 352, in main_export
    export_from_model(
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\exporters\openvino\convert.py", line 657, in export_from_model
    export_models(
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\exporters\openvino\convert.py", line 476, in export_models
    export(
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\exporters\openvino\convert.py", line 149, in export
    return export_pytorch(
           ^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\exporters\openvino\convert.py", line 380, in export_pytorch
    return export_pytorch_via_onnx(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\exporters\openvino\convert.py", line 249, in export_pytorch_via_onnx
    input_names, output_names = export_pytorch_to_onnx(
                                ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\exporters\onnx\convert.py", line 577, in export_pytorch
    onnx_export(
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\onnx\utils.py", line 516, in export
    _export(
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\onnx\utils.py", line 1612, in _export
    graph, params_dict, torch_out = _model_to_graph(
                                    ^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\onnx\utils.py", line 1134, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\onnx\utils.py", line 1010, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\onnx\utils.py", line 914, in _trace_and_get_graph_from_model
    trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\jit\_trace.py", line 1310, in _get_trace_graph
    outs = ONNXTracedModule(
           ^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\nncf\torch\dynamic_graph\wrappers.py", line 146, in wrapped
    return module_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\jit\_trace.py", line 138, in forward
    graph, out = torch._C._create_graph_by_tracing(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\jit\_trace.py", line 129, in wrapper
    outs.append(self.inner(*trace_inputs))
                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\nncf\torch\dynamic_graph\wrappers.py", line 146, in wrapped
    return module_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1522, in _slow_forward
    result = self.forward(*input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\exporters\onnx\model_patcher.py", line 152, in patched_forward
    outputs = self.orig_forward(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\.cache\huggingface\modules\transformers_modules\internlm\internlm2-chat-7b\70e6cdc9643ce7e3d9a369fb984dc5f1a1b2cec6\modeling_internlm2.py", line 1204, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\nncf\torch\dynamic_graph\wrappers.py", line 146, in wrapped
    return module_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1522, in _slow_forward
    result = self.forward(*input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\.cache\huggingface\modules\transformers_modules\internlm\internlm2-chat-7b\70e6cdc9643ce7e3d9a369fb984dc5f1a1b2cec6\modeling_internlm2.py", line 1004, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\nncf\torch\dynamic_graph\wrappers.py", line 146, in wrapped
    return module_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1522, in _slow_forward
    result = self.forward(*input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\.cache\huggingface\modules\transformers_modules\internlm\internlm2-chat-7b\70e6cdc9643ce7e3d9a369fb984dc5f1a1b2cec6\modeling_internlm2.py", line 738, in forward
    hidden_states, self_attn_weights, present_key_value = self.attention(
                                                          ^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\nncf\torch\dynamic_graph\wrappers.py", line 146, in wrapped
    return module_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\torch\nn\modules\module.py", line 1522, in _slow_forward
    result = self.forward(*input, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Local_Admin\miniforge3\envs\ov-nightly-benchmarking\Lib\site-packages\optimum\exporters\openvino\model_patcher.py", line 1117, in _internlm2_attention_forward
    kv_seq_len += past_key_value[0].shape[-2]
                  ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'tuple' object has no attribute 'shape'

[eaidova]

Hello, @lumurillo, I prepared PR with fix https://github.com/huggingface/optimum-intel/pull/814

[lumurillo]

Thanks @eaidova, that solved the issue!