search

alibaba / rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
Apache License 2.0
500 stars 47 forks source link

`Illegal instruction` error when running version 0.2.0 #89

Closed frankang closed 1 month ago

frankang commented 1 month ago

Hi, I encountered an error when trying to start the server using version 0.2.0. Specifically, I tried both the 0.2.0 package from the releases section with cuda 12.1 and a package I built from the latest code with CUDA 11.8 (but the package I built is only about 95MB).

When running the command (MODEL and TOKENIZER info omitted) FT_SERVER_TEST=1 python3 -m maga_transformer.start_server, I got the following error:

[process-101471][root][07/17/2024 15:26:39][__init__.py:<module>():14][INFO] init logger end
so path: /home/user/miniconda3/envs/pytorch2.1_cuda11.8/lib/python3.10/site-packages/maga_transformer/libs
Illegal instruction

The Illegal instruction error also happened on the 0.2.0 wheel with the cuda12.1 environment.

I previously tried the same command and model with the v0.1.2 ~ v0.1.4 rtp-llm version with no errors. I'm using pytorch 2.1.0 and NVIDIA 3090.

Any help would be greatly appreciated. Thanks!

netaddi commented 1 month ago

Hi, this is a known issue caused by a library from intel, which used avx512 instruction and lead to core dump if your cpu does not support avx512 instruction set. We are going to disable this library in next release, and for now please comment from here https://github.com/alibaba/rtp-llm/blob/957b38ea88ff262c38abf9de03a428a20dae47ce/maga_transformer/ops/__init__.py#L39 to line 51.

frankang commented 1 month ago

Thanks. Though I still got other errors after commenting out the above section, I don't have enough time debugging into it.