Closed Mingyu-Wei closed 10 months ago
I fix these issues in this #PR , these are some details:
pip install bigdl-llm[all]
command with pip install --pre --upgrade bigdl-llm[all]
load_in_low_bit
options, for example for CPU
Currently, `load_in_low_bit` supports options `'sym_int4'`, `'asym_int4'`, `'sym_int5'`, `'asym_int5'` or `'sym_int8'`, in which 'sym' and 'asym' differentiate between symmetric and asymmetric quantization. `'nf3'` and `'nf4'` stand for normalFloat quantizations . Floating point precision `'fp4'`, `'fp8'`, `'fp16'` and Option `'mixed_4bit'` are also supported.
BigDL LLM package installation: I notice that in some chapters the package installation suggestion is :
However, I also see:
in some files. Is it necessary to unify this command?
link outdated: The links at the end of chapter 1 are all outdated.
All the links above have already been changed, so they all lead to 404 Page Not Found. For example, the current address of Llama2 in the tutorial is: https://github.com/intel-analytics/BigDL/blob/main/python/llm/example/transformers/native_int4, but the folder structure has been updated and it should be https://github.com/intel-analytics/BigDL/tree/main/python/llm/example/CPU/HF-Transformers-AutoModels/Model/llama2 now.
load_in_low_bit options outdated: In chapter 5/6, the load_in_low_bit options are:
The latest version of bigdl-llm 2.4.0 supports sym_int4, asym_int4, sym_int5, asym_int5, sym_int8, nf3, nf4, fp4, fp8, fp16 or mixed_4bit. And for GPU in chapter 6, the supported options are: sym_int4, asym_int4, sym_int5, asym_int5, sym_int8, nf3, nf4, fp4, fp8, fp16, mixed_fp4 or mixed_fp8
Sample audio files in chapter 5.2 deprecated:
The common voice dataset is deprecated and will be deleted soon according to their hugging face. As for the audio files(audio_en.mp3/audio_zh.mp3) downloaded in the wget command, these files are already removed from hugging face. Using these files will lead to EOF error when running the sample code in this section.
In this image, the command
source /opt/intel/oneapi/setvars.sh
is listed as a recommendation for Intel GPU acceleration. However, based on my own knowledge and experience, this command is mandatory and should be used whenever a new terminal session is created. Otherwise we might encounter this errorOSError: libmkl_intel_lp64.so.2: cannot open shared object file: No such file or directory
. This is not exactly an error, but I believe it might be better to highlight this command in the tutorial, either in README.md or in 6_1_GPU_Llama2-7B.md