Closed aadarsh-ram closed 9 months ago
@vinx13 is working on a fix
Just wanted to follow up with you, @vinx13, is this issue fixed already?
I'm working on a fix, we need to update dlight rules to choose appropriate tile size
Hi @aadarsh-ram, we suppose the issue has been already fixed. Would you like to checkout the latest TVM Unity and try again? You are welcome to reopen the issue if there are further problems.
@MasterJH5574 I'll try the compilation process again and let you know. Thank you!
Hey @MasterJH5574, I'm getting the following error. I built TVM-Unity as per the instructions here.
(myenv) aadarsh@AAD-HPLAP:~/src/mlc-llm$ python3 -m mlc_llm.build --hf-path pankajmathur/orca_mini_3b --target vulkan --quantization q4f16_1
Weights exist at dist/models/orca_mini_3b, skipping download.
Using path "dist/models/orca_mini_3b" for model "orca_mini_3b"
Target configured: vulkan -keys=vulkan,gpu -max_num_threads=256 -max_shared_memory_per_block=32768 -max_threads_per_block=256 -supports_16bit_buffer=1 -supports_8bit_buffer=1 -supports_float16=1 -supports_float32=1 -supports_int16=1 -supports_int32=1 -supports_int8=1 -supports_storage_buffer_storage_class=1 -thread_warp_size=1
Failed to detect local GPU, falling back to CPU as a target
Automatically using target for weight quantization: llvm -keys=cpu
Get old param: 0%| | 0/161 [00:00<?, ?tensors/sStart computing and quantizing weights... This may take a while. | 0/267 [00:00<?, ?tensors/s]
Get old param: 1%|▍ | 1/161 [00:00<02:16, 1.17tensors/s]
Set new param: 0%|▏ | 1/267 [00:00<03:46, 1.17tensors/s]
Get old param: 98%|███████████████████████████████████████████████████████████▊ | 158/161 [03:47<00:04, 1.55s/tensorsFinish computing and quantizing weights.█████████████████████████████████████████▊| 266/267 [03:52<00:00, 1.01tensors/s]
Total param size: 1.7960131168365479 GB
Start storing to cache dist/orca_mini_3b-q4f16_1/params
[0267/0267] saving param_266
All finished, 67 total shards committed, record saved to dist/orca_mini_3b-q4f16_1/params/ndarray-cache.json
Finish exporting chat config to dist/orca_mini_3b-q4f16_1/params/mlc-chat-config.json
Save a cached module to dist/orca_mini_3b-q4f16_1/mod_cache_before_build.pkl.
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/aadarsh/src/mlc-llm/mlc_llm/build.py", line 46, in <module>
main()
File "/home/aadarsh/src/mlc-llm/mlc_llm/build.py", line 42, in main
core.build_model_from_args(parsed_args)
File "/home/aadarsh/src/mlc-llm/mlc_llm/core.py", line 686, in build_model_from_args
build(mod, args)
File "/home/aadarsh/src/mlc-llm/mlc_llm/core.py", line 578, in build
ex = relax.build(mod_deploy, args.target, system_lib=args.system_lib)
File "/home/aadarsh/src/tvm-unity/python/tvm/relax/vm_build.py", line 343, in build
return _vmlink(builder, target, tir_mod, ext_libs, params, system_lib=system_lib)
File "/home/aadarsh/src/tvm-unity/python/tvm/relax/vm_build.py", line 242, in _vmlink
lib = tvm.build(
File "/home/aadarsh/src/tvm-unity/python/tvm/driver/build_module.py", line 281, in build
rt_mod_host = _driver_ffi.tir_to_runtime(annotated_mods, target_host)
File "/home/aadarsh/src/tvm-unity/python/tvm/_ffi/_ctypes/packed_func.py", line 239, in __call__
raise_last_ffi_error()
File "/home/aadarsh/src/tvm-unity/python/tvm/_ffi/base.py", line 481, in raise_last_ffi_error
raise py_err
File "/home/aadarsh/src/tvm-unity/src/driver/driver_api.cc", line 527, in operator()
return TIRToRuntime(inputs_arg, host_target);
File "/home/aadarsh/src/tvm-unity/src/driver/driver_api.cc", line 510, in tvm::TIRToRuntime(tvm::runtime::Map<tvm::Target, tvm::IRModule, void, void> const&, tvm::Target const&)
device_modules.push_back(codegen::Build(device_mod, it.first));
File "/home/aadarsh/src/tvm-unity/src/target/codegen.cc", line 72, in tvm::codegen::Build(tvm::IRModule, tvm::Target)
ICHECK(bf != nullptr) << build_f_name << " is not enabled";
tvm.error.InternalError: Traceback (most recent call last):
2: operator()
at /home/aadarsh/src/tvm-unity/src/driver/driver_api.cc:527
1: tvm::TIRToRuntime(tvm::runtime::Map<tvm::Target, tvm::IRModule, void, void> const&, tvm::Target const&)
at /home/aadarsh/src/tvm-unity/src/driver/driver_api.cc:510
0: tvm::codegen::Build(tvm::IRModule, tvm::Target)
at /home/aadarsh/src/tvm-unity/src/target/codegen.cc:72
File "/home/aadarsh/src/tvm-unity/src/target/codegen.cc", line 72
InternalError: Check failed: (bf != nullptr) is false: target.build.vulkan is not enabled
Hi @aadarsh-ram, you mentioned you built TVM Unity from source. Could you help check and make sure set(USE_VULKAN ON)
is set in build/config.cmake
? Following the instructions here https://llm.mlc.ai/docs/install/tvm.html#option-2-build-from-source, the commands would be
# Now at path/to/tvm
cd build
cp ../cmake/config.cmake .
# controls default compilation flags
echo "set(CMAKE_BUILD_TYPE RelWithDebInfo)" >> config.cmake
# LLVM is a must dependency
echo "set(USE_LLVM \"llvm-config --ignore-libllvm --link-static\")" >> config.cmake
echo "set(HIDE_PRIVATE_SYMBOLS ON)" >> config.cmake
echo "set(USE_VULKAN ON)" >> config.cmake
After finish the cmake configuration we can build TVM again. It is then expected to have Vulkan enabled.
And here is an easy check in Python to validate if vulkan is enabled:
>>> import tvm
>>> tvm.vulkan()
vulkan(0)
If it displays vulkan(0)
as output, then Vulkan is successfully enabled.
Yep, perfect! It worked. Thank you for your help!
Glad to hear it works!
🐛 Bug
I set up mlc-llm and tried to compile Orca-Mini 3B. I am getting the following error:
Please let me know further steps to rectify this error.
Environment
python -c "import tvm; print('\n'.join(f'{k}: {v}' for k, v in tvm.support.libinfo().items()))"
, applicable if you compile models):