search

intel / intel-extension-for-pytorch

A Python package for extending the official PyTorch that can easily obtain performance on Intel platform
Apache License 2.0
1.49k stars 231 forks source link

Can ipex support general llm like defog/sqlcoder #413

Open RenyanDiao opened 11 months ago

RenyanDiao commented 11 months ago

Describe the issue

model = ipex._optimize_transformers(model.eval(), dtype=torch.bfloat16, inplace=True)

image

jingxu10 commented 11 months ago

No, currently only models mentioned in the screenshot are optimized.

RenyanDiao commented 11 months ago

No, currently only models mentioned in the screenshot are optimized.

Do you have plans to support more models recently?

jingxu10 commented 11 months ago

Is there any reproducer that we can use to investigate this crash issue? Models outside the list should be able to run, but will not be optimized.

RenyanDiao commented 11 months ago

Is there any reproducer that we can use to investigate this crash issue? Models outside the list should be able to run, but will not be optimized.

https://github.com/defog-ai/sqlcoder/tree/main yes, it can run while just performs as the same as using official torch. The latency is too long for us.