LangChain-Chatchat shows RuntimeError: could not create a primitive

[MYaoBQ]

i'm using NUC with win 11 on ARC 770 (driver version: 31.0.101.5379) and UHD Graphics 770 (iGPU). Shows error (RuntimeError: could not create a primitive) by following up the latest instruction of LangChain-ChatChat. please see below capture. ARC 770 is on device 0.

full capture of the issue.txt

[Oscilloscope98]

Hi @MYaoBQ,

This may due to a bug from IPEX when iGPU and dGPU are both available, and especially when dGPU is not recognized as 'xpu:0'.

To solve this problem and run on ARC A770, you may try to disable UHD Graphics 770 on your machine first and have a try again :)

You could do that through

Device manager (设备管理器) -> Display adapter (显示适配器) -> UHD Graphics 770 - right click (右击) -> Disable device (禁用设备)

Please let us know for any further problems :)

[MYaoBQ]

Yuwen, Just tried after disable the iGPU (UHD Graphics 770), but it hangs on “converting the current model to sym_int4…” like forever, I have to kill the process by using Ctl+C. any advice? Thanks,

@.***

[A close up of a sign Description automatically generated] Michael Yao Technical Sales Specialist for Enterprise Client | Sales and Marketing

M +8613501999436 @.**@.> Intel Corporation | intel.comhttp://intel.com/

From: Yuwen Hu @.> Sent: Monday, April 1, 2024 2:47 PM To: intel-analytics/ipex-llm @.> Cc: Yao, Michael BQ @.>; Mention @.> Subject: Re: [intel-analytics/ipex-llm] LangChain-Chatchat shows RuntimeError: could not create a primitive (Issue #10605)

Hi @MYaoBQhttps://github.com/MYaoBQ,

This may due to a bug from IPEX when iGPU and dGPU are both available, and especially when dGPU is not recognized as 'xpu:0'.

To solve this bug and run on ARC A770, you may try to disable UHD Graphics 770 on your machine first and have a try again :)

You could do that through

Device manager (设备管理器) -> Display adapter (显示适配器) -> UHD Graphics 770 -> Disable device (禁用设备)

and have a try again? :)

— Reply to this email directly, view it on GitHubhttps://github.com/intel-analytics/ipex-llm/issues/10605#issuecomment-2029260371, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BHPRSHJFBPMKFU73ST7XLLTY3D7GRAVCNFSM6AAAAABFQ5UT5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRZGI3DAMZXGE. You are receiving this because you were mentioned.Message ID: @.**@.>>

[Oscilloscope98]

Hi @MYaoBQ,

There are several things you could have a try:

1. Restart the machine after disabling iGPU
2. No need to set ONEAPI_DEVICE_SELECTOR any more as only one GPU is available :)

[MYaoBQ]

Thanks, it works after disabling the iGPU and reboot.

When testing RAG, sometimes it shows below error message “API 通信错误”, please see below example.

No issue when ask “what’s ArrowLake”. But error pop up when ask “what is ARL”.

What’s the reason behind? Both “Arrow Lake” and “ARL” are in the documents that uploaded. Thanks.

@.***

[A close up of a sign Description automatically generated] Michael Yao Technical Sales Specialist for Enterprise Client | Sales and Marketing

M +8613501999436 @.**@.> Intel Corporation | intel.comhttp://intel.com/

From: Yuwen Hu @.> Sent: Monday, April 1, 2024 6:23 PM To: intel-analytics/ipex-llm @.> Cc: Yao, Michael BQ @.>; Mention @.> Subject: Re: [intel-analytics/ipex-llm] LangChain-Chatchat shows RuntimeError: could not create a primitive (Issue #10605)

Hi @MYaoBQhttps://github.com/MYaoBQ,

There are several things you could have a try:

1. Restart the machine after disabling iGPU
2. No need to set ONEAPI_DEVICE_SELECTOR any more as only one GPU is available :)

— Reply to this email directly, view it on GitHubhttps://github.com/intel-analytics/ipex-llm/issues/10605#issuecomment-2029538912, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BHPRSHI6CFVTVTYZBKCGRLDY3EYPLAVCNFSM6AAAAABFQ5UT5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRZGUZTQOJRGI. You are receiving this because you were mentioned.Message ID: @.**@.>>

[Oscilloscope98]

Hi @MYaoBQ ,

We have recently fixed this 通信错误 error, please kindly update to our latest code and have a try again :)

Besides, for Arc A770, here are the recommended configuations:

set USE_XETLA=OFF
set SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
set SYCL_CACHE_PERSISTENT=1
set BIGDL_QUANTIZE_KV_CACHE=1
set BIGDL_LLM_XMX_DISABLED=1

set no_proxy=localhost,127.0.0.1
set BIGDL_IMPORT_IPEX=0

python startup.py -a

Please let us know for any further problems :)

[violet17]

Hi @MYaoBQ ,

We have recently fixed this 通信错误 error, please kindly update to our latest code and have a try again :)

Besides, for Arc A770, here are the recommended configuations:

set USE_XETLA=OFF
set SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
set SYCL_CACHE_PERSISTENT=1
set BIGDL_QUANTIZE_KV_CACHE=1
set BIGDL_LLM_XMX_DISABLED=1

set no_proxy=localhost,127.0.0.1
set BIGDL_IMPORT_IPEX=0

python startup.py -a

Please let us know for any further problems :)

Base on your reply, I try it on Arc dGPU, but get error. set BIGDL_IMPORT_IPEX=0 means that IPEX won't be imported, right?

[Oscilloscope98]

Hi @violet17,

We have currently updated our Windows guide on Arc A-Series, it is recommended to follow our guide for Langchain-chatchat deployment on WIndows Arc A-Series.

Besides, set BIGDL_IMPORT_IPEX=0 means not automatically importing ipex, but we manually did that in our Langchain-chatchat support code.

Please let us know for any further problems :)

[violet17]

Hi @Oscilloscope98 , thank you for your reply. I fix could not create a primitive with disable iGPU when using dGPU for inference instead of using set BIGDL_IMPORT_IPEX=0.

[Oscilloscope98]

Hi @Oscilloscope98 , thank you for your reply. I fix could not create a primitive with disable iGPU when using dGPU for inference instead of using set BIGDL_IMPORT_IPEX=0.

Hi @violet17,

I'm glad you solved the problem :) Just some clarification that set BIGDL_IMPORT_IPEX=0 is not for resolving the could not create a primitive issue. It is related to this issue.

Please let me know for any further problems :)