Open p-wysocki opened 6 months ago
Hi OpenVino developers, I'm interested in GSOC this summer. Could I take this task please? it is a good chance to get familiar with the codebase and development workflow.
Thanks
Hello @mengbingrock! Thanks for taking a look, I assigned you. Please let us know if you have any questions. :)
Hello Developers, I met some problem during export:
[ WARNING ] Cannot apply model.to_bettertransformer because of the exception:
The model type baichuan is not yet supported to be used with BetterTransformer. Feel free to open an issue at https://github.com/huggingface/optimum/issues if you would like this model type to be supported. Currently supported models are: dict_keys(['albert', 'bark', 'bart', 'bert', 'bert-generation', 'blenderbot', 'bloom', 'camembert', 'blip-2', 'clip', 'codegen', 'data2vec-text', 'deit', 'distilbert', 'electra', 'ernie', 'fsmt', 'gpt2', 'gptj', 'gpt_neo', 'gpt_neox', 'hubert', 'layoutlm', 'm2m_100', 'marian', 'markuplm', 'mbart', 'opt', 'pegasus', 'rembert', 'prophetnet', 'roberta', 'roc_bert', 'roformer', 'splinter', 'tapas', 't5', 'vilt', 'vit', 'vit_mae', 'vit_msn', 'wav2vec2', 'xlm-roberta', 'yolos', 'stablelm_epoch', 'aquila', 'codegen2']).. Usage model with stateful=True may be non-effective if model does not contain torch.functional.scaled_dot_product_attention
Overriding 1 configuration item(s)
- use_cache -> True
./build/greedy_causal_lm ./Baichuan2-7B-Chat/pytorch/dldt/FP16/ "Why is the Sun yellow?"
Exception from src/inference/src/infer_request.cpp:196:
Check '::getPort(port, name, {_impl->get_inputs(), _impl->get_outputs()})' failed at src/inference/src/infer_request.cpp:198:
Port for tensor name position_ids was not found.
The position_ids indeed in not shown in openvino_model.xml. Do I need to check the intel.optimum for solution?
Thanks
@pavel-esir
The position_ids indeed in not shown in openvino_model.xml. Do I need to check the intel.optimum for solution?
hi @mengbingrock, thanks for you analysis! Yes, looking to intel.optimum might help to find why position_ids are not displayed in IR. You can but the breakpoing or a pring right before forward
method to see what argument are fed into network inputs.
I will also take a look what input are fed to forward
in the very original HF repo, but i bit later
Thank you for your reply, @pavel-esir Before forwarding, the greedy_causal_lm.cpp line 78 would check node name position_ids is present or not. And it failed to find it.
I thought the ir is responsible for this error, and it is generated from convert.py,
def convert_optimum_causallm_base(model, args, model_config=None, compress_only=False):
which is calling optimum.intel. This is a customized model and it need special configuration during exporting. I noticed that there is works already exported it the onnx with correct input name. I'm looking at its convertion code to understand it.
ref: https://github.com/wangzhaode/llm-export/releases/tag/baichuan2-7b-chat-onnx
Hi @pavel-esir, I ran this command directly, then it produce the similiar xml as last time.
optimum-cli export openvino --trust-remote-code --model ~/.cache/huggingface/hub/models--baichuan-inc--Baichuan2-7B-Chat/snapshots/ea66ced17780ca3db39bc9f8aa601d8463db3da5 --task text-generation-with-past bcaichuan
No position_ids is present, missing this parameter in IR.
Where should I look at next? Thank your for your guidance.
@mengbingrock thx for the update. I'm right debugging conversion in optimum to see why position_ids disappeared
@mengbingrock i finally managed to get IR with position_ids
openvino_model.xml.txt.
In order to do so, here https://github.com/huggingface/optimum-intel/blob/main/optimum/exporters/openvino/model_configs.py#L77
TextDecoderOnnxConfig
should be changed to TextDecoderWithPositionIdsOnnxConfig
.
Soon we will open PR for that, but meantime as a workaround you can modify your file locally
venv_path/site-packages/optimum/exporters/openvino/model_configs.py
Really appreciate your work on this @pavel-esir ! Next time I'll try to find the root cause myself to save your valuable time. I've drafted the PR, but it is after your commit to optimum.
Context
This task regards enabling tests for baichuan2-7b-chat. You can find more details under openvino_notebooks LLM chatbot README.md.
Please ask general questions in the main issue at https://github.com/openvinotoolkit/openvino.genai/issues/259
What needs to be done?
Described in the main Discussion issue at: https://github.com/openvinotoolkit/openvino.genai/issues/259
Example Pull Requests
Described in the main Discussion issue at: https://github.com/openvinotoolkit/openvino.genai/issues/259
Resources
Contact points
Described in the main Discussion issue at: https://github.com/openvinotoolkit/openvino.genai/issues/259
Ticket
No response