Error When Implementing Mali GPU Acceleration on OrangePi5 with mlc-llm

[Yusepp]

Following the tutorial, I set up mlc-llm on my OrangePi5 with Mali GPU acceleration via OpenCL. Everything was smooth until I encountered an error. I've re-downloaded the Mali libraries (versions below) multiple times, but the error persists. Could the libraries be corrupted?

Library versions in use:

• RedPajama-INCITE-Chat-3B-v1-q4f16_1
• RedPajama-INCITE-Chat-3B-v1-q4f16_1-mali.so

Any advice on resolving this would be appreciated.

arm_release_ver: g13p0-01eac0, rk_so_ver: 3
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '7'.
Traceback (most recent call last):
  File "/home/yusepp/Desktop/test.py", line 6, in <module>
    cm = ChatModule(model=models+"/RedPajama-INCITE-Chat-3B-v1-q4f16_1",
  File "/home/yusepp/mlc-llm/python/mlc_chat/chat_module.py", line 842, in __init__
    self._reload(self.model_lib_path, self.model_path, user_chat_config_json_str)
  File "/home/yusepp/mlc-llm/python/mlc_chat/chat_module.py", line 1056, in _reload
    self._reload_func(lib, model_path, app_config_json)
  File "/home/yusepp/tvm_unity/python/tvm/_ffi/_ctypes/packed_func.py", line 239, in __call__
    raise_last_ffi_error()
  File "/home/yusepp/tvm_unity/python/tvm/_ffi/base.py", line 481, in raise_last_ffi_error
    raise py_err
  File "/home/yusepp/Desktop/tvm_unity/src/runtime/relax_vm/ndarray_cache_support.cc", line 255, in tvm::runtime::relax_vm::NDArrayCache::Load(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)
ValueError: Traceback (most recent call last):
  3: 0x0000ffff63d3ae9b
  2: 0x0000ffff63d3ac23
  1: 0x0000ffff63d392bf
  0: tvm::runtime::relax_vm::NDArrayCache::Load(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)
        at /home/yusepp/Desktop/tvm_unity/src/runtime/relax_vm/ndarray_cache_support.cc:255
  4: 0x0000ffff63d3ae9b
  3: 0x0000ffff63d3ac23
  2: 0x0000ffff63d392bf
  1: tvm::runtime::relax_vm::NDArrayCache::Load(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)
        at /home/yusepp/Desktop/tvm_unity/src/runtime/relax_vm/ndarray_cache_support.cc:253
  0: tvm::runtime::relax_vm::NDArrayCacheMetadata::FileRecord::Load(DLDevice, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, tvm::runtime::Optional<tvm::runtime::NDArray>*) const
        at /home/yusepp/Desktop/tvm_unity/src/runtime/relax_vm/ndarray_cache_support.cc:193
  File "/home/yusepp/Desktop/tvm_unity/src/runtime/relax_vm/ndarray_cache_support.cc", line 255
ValueError: Error when loading parameters from params_shard_0.bin: [20:19:57] /home/yusepp/Desktop/tvm_unity/src/runtime/relax_vm/ndarray_cache_support.cc:193: Check failed: this->nbytes == raw_data_buffer->length() (64552960 vs. 133) : ValueError: Encountered an corrupted parameter shard. It means it is not downloaded completely or downloading is interrupted. Please try to download again.

[louis030195]

@Yusepp had this too, installed git lfs and re downloaded stuff and it worked

[chaitanyakrishna1248]

Hi @Yusepp ,I am also following the OrangePi5 tutorial, could you please tell me how you installed mlc chat? I followed the tutorials but when I run the python file it says there is no directory called mlc chat.