Open Jansper-x opened 12 months ago
Sorry, I cannot reproduce the second issue :disappointed_relieved:
This is my environment:
#/bin/bash
export DOCKER_IMAGE=intelanalytics/bigdl-ppml-trusted-bigdl-llm-gramine-ref:2.4.0-SNAPSHOT
export MODEL_PATH="/home/llm/models/"
sudo docker run -itd \
--net=host \
--cpuset-cpus="5-20" \
--cpuset-mems="0" \
--memory="32G" \
--name=gc-debug \
-v $MODEL_PATH:/ppml/models \
--shm-size="16g" \
$DOCKER_IMAGE
Then, docker exec -it gc-debug bash
I tried both chatglm-6b
and chatglm2-6b
models, and they both worked fine in my environment:
Do you use our latest intelanalytics/bigdl-ppml-trusted-bigdl-llm-gramine-ref:2.4.0-SNAPSHOT
image? :eyes:
You can also add a print
statement at /ppml/FastChat/fastchat/model/model_adapter.py
L545 to ensure that the patched ChatGLM model adapter is used.
For the first issue, @hzjane can you help to fix this issue?
The first issue may be caused by changes in the BigDL-LLM version. To address this, You can use the latest Dockerfile to build.
Sorry, I cannot reproduce the second issue 😥
This is my environment:
#/bin/bash export DOCKER_IMAGE=intelanalytics/bigdl-ppml-trusted-bigdl-llm-gramine-ref:2.4.0-SNAPSHOT export MODEL_PATH="/home/llm/models/" sudo docker run -itd \ --net=host \ --cpuset-cpus="5-20" \ --cpuset-mems="0" \ --memory="32G" \ --name=gc-debug \ -v $MODEL_PATH:/ppml/models \ --shm-size="16g" \ $DOCKER_IMAGE
Then,
docker exec -it gc-debug bash
I tried both
chatglm-6b
andchatglm2-6b
models, and they both worked fine in my environment:
![]()
Do you use our latest
intelanalytics/bigdl-ppml-trusted-bigdl-llm-gramine-ref:2.4.0-SNAPSHOT
image? 👀You can also add a
/ppml/FastChat/fastchat/model/model_adapter.py
L545 to ensure that the patched ChatGLM model adapter is used.For the first issue, @hzjane can you help to fix this issue?
Thanks, I have solved this problem. The reason is that I mistakenly named the model folder of chatglm2-6b as chatglm-6b before. This may cause the error in model loading. I have changed the folder name back to chatglm2-6b and it loads and runs normally. Now I can run the model and perform inference in Docker.
I noticed that BigDL accelerates the computation of inference by using 512 AVX. But when I run the Fastchat on my local host without bigdl and in the docker images with bigdl.The inference times I got are not much different. Sometimes it takes longer in the docker. For example: On my local platform this question takes 36 seconds, but in the docker images it takes 62 seconds.
Does this require additional configuration? I have chack my cpu suppport the AVX512.
Do you use the same core/mem configuration when you run in container and local machine?
Besides, please try --cpuset-cpus
option instead of --cpus
We recommend to use source bigdl-nano-init -t
to setup the environment, try do the following:
source bigdl-nano-init -t
# bigdl-nano-init may set wrong number of cores in container
export OMP_NUM_THREADS="YOUR_CORE_NUM"
Sorry, I cannot reproduce the second issue 😥 This is my environment:
#/bin/bash export DOCKER_IMAGE=intelanalytics/bigdl-ppml-trusted-bigdl-llm-gramine-ref:2.4.0-SNAPSHOT export MODEL_PATH="/home/llm/models/" sudo docker run -itd \ --net=host \ --cpuset-cpus="5-20" \ --cpuset-mems="0" \ --memory="32G" \ --name=gc-debug \ -v $MODEL_PATH:/ppml/models \ --shm-size="16g" \ $DOCKER_IMAGE
Then,
docker exec -it gc-debug bash
I tried bothchatglm-6b
andchatglm2-6b
models, and they both worked fine in my environment:![]()
Do you use our latest
intelanalytics/bigdl-ppml-trusted-bigdl-llm-gramine-ref:2.4.0-SNAPSHOT
image? 👀 You can also add a/ppml/FastChat/fastchat/model/model_adapter.py
L545 to ensure that the patched ChatGLM model adapter is used. For the first issue, @hzjane can you help to fix this issue?Thanks, I have solved this problem. The reason is that I mistakenly named the model folder of chatglm2-6b as chatglm-6b before. This may cause the error in model loading. I have changed the folder name back to chatglm2-6b and it loads and runs normally. Now I can run the model and perform inference in Docker.
I noticed that BigDL accelerates the computation of inference by using 512 AVX. But when I run the Fastchat on my local host without bigdl and in the docker images with bigdl.The inference times I got are not much different. Sometimes it takes longer in the docker. For example: On my local platform this question takes 36 seconds, but in the docker images it takes 62 seconds.
![]()
Does this require additional configuration? I have chack my cpu suppport the AVX512.
I increased the numbers of cores and memory and the inference was faster inside the container than outside. I want to learn more information about how bigdl-llm uses avx for acceleration. How is this acceleration implemented ? Where is this part of the code? Is there any change or patching on pytorch?
I increased the numbers of cores and memory and the inference was faster inside the container than outside. I want to learn more information about how bigdl-llm uses avx for acceleration. How is this acceleration implemented ? Where is this part of the code? Is there any change or patching on pytorch?
bigdl-llm will automatically apply intel specific optimizations (including amx/vnni/avx/etc.) to LLM; no change/patching to pytorch is needed
I try to use trusted-bigdl-llm to run the Fastchat. When I build the base image,I got the patching failed error:
My Dockerfile:
Then I use the public reference image provided by BigDL PPML intelanalytics/bigdl-ppml-trusted-bigdl-llm-gramine-ref:2.4.0-SNAPSHOT to run the fastchat
But I got this error: