search

intel / intel-extension-for-pytorch

A Python package for extending the official PyTorch that can easily obtain performance on Intel platform
Apache License 2.0
1.64k stars 254 forks source link

Can ipex support general llm like defog/sqlcoder #413

Open RenyanDiao opened 1 year ago

RenyanDiao commented 1 year ago

Describe the issue

model = ipex._optimize_transformers(model.eval(), dtype=torch.bfloat16, inplace=True)

image

jingxu10 commented 1 year ago

No, currently only models mentioned in the screenshot are optimized.

RenyanDiao commented 1 year ago

No, currently only models mentioned in the screenshot are optimized.

Do you have plans to support more models recently?

jingxu10 commented 1 year ago

Is there any reproducer that we can use to investigate this crash issue? Models outside the list should be able to run, but will not be optimized.

RenyanDiao commented 1 year ago

Is there any reproducer that we can use to investigate this crash issue? Models outside the list should be able to run, but will not be optimized.

https://github.com/defog-ai/sqlcoder/tree/main yes, it can run while just performs as the same as using official torch. The latency is too long for us.