search

mlc-ai / mlc-llm

Universal LLM Deployment Engine with ML Compilation
https://llm.mlc.ai/
Apache License 2.0
19.08k stars 1.56k forks source link

[Bug] AttributeError: function 'TVMGetLastPythonError' not found. Did you mean: 'TVMAPISetLastPythonError'? #1112

Closed David-Sharma closed 11 months ago

David-Sharma commented 1 year ago

🐛 Bug

(base) C:\Users\dmsha\dev\mlc>python -m mlc_llm.build --model Llama-2-7b-chat-hf --target vulkan --quantiz ation q4f16_1 --llvm-mingw path/to/llvm-mingw ** Compiling models under Windows 11 has not been an issue for me. This started about 1 week ago and I have not been able to resolve. A reload has not cleared this problem.

To Reproduce

Steps to reproduce the behavior:

1.(base) C:\Users\dmsha\dev\mlc>python -m mlc_llm.build --model Llama-2-7b-chat-hf --target vulkan --quantiz ation q4f16_1 --llvm-mingw path/to/llvm-mingw 1. 1.

Using path "dist\models\Llama-2-7b-chat-hf" for model "Llama-2-7b-chat-hf" Target configured: vulkan -keys=vulkan,gpu -max_num_threads=256 -max_shared_memory_per_block=32768 -max_threads_per_block=256 -supports_16bit_buffer=1 -supports_8bit_buffer=1 -supports_float16=1 -supports_float32=1 -supports_int16=1 -supports_int32=1 -supports_int8=1 -supports_storage_buffer_storage_class=1 -thread_warp_size=1

The following 2 lines reproduces themselves many times: [14:37:25] D:\a\package\package\tvm\src\node\reflection.cc:109: AttributeError: relax.expr.Var object has no attributed shard_dim Stack trace not available when DMLC_LOG_STACK_TRACE is disabled at compile time.

[14:37:25] D:\a\package\package\tvm\src\node\reflection.cc:109: AttributeError: relax.expr.Var object has no attributed shard_strategy Stack trace not available when DMLC_LOG_STACK_TRACE is disabled at compile time.

[14:37:27] D:\a\package\package\tvm\src\relax\ir\expr.cc:174: Check failed: index < tuple_info->fields.size() (197 vs. 197) : Index out of bounds: Tuple params is of size 197, and cannot be accessed with index 197 Stack trace not available when DMLC_LOG_STACK_TRACE is disabled at compile time.

Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "C:\Users\dmsha\miniconda3\Lib\site-packages\mlc_llm\build.py", line 46, in main() File "C:\Users\dmsha\miniconda3\Lib\site-packages\mlc_llm\build.py", line 42, in main core.build_model_from_args(parsed_args) File "C:\Users\dmsha\miniconda3\Lib\site-packages\mlc_llm\core.py", line 648, in build_model_from_args new_params = utils.convert_weights(param_manager, params, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\dmsha\miniconda3\Lib\site-packages\mlc_llm\utils.py", line 229, in convert_weights mod_transform = relax.transform.LazyTransformParams()(mod_transform) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\dmsha\miniconda3\Lib\site-packages\tvm\ir\transform.py", line 238, in call return _ffi_transform_api.RunPass(self, mod) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\dmsha\miniconda3\Lib\site-packages\tvm_ffi_ctypes\packed_func.py", line 239, in call raise_last_ffi_error() File "C:\Users\dmsha\miniconda3\Lib\site-packages\tvm_ffi\base.py", line 415, in raise_last_ffi_error _LIB.TVMGetLastPythonError.restype = ctypes.c_void_p ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\dmsha\miniconda3\Lib\ctypes__init.py", line 389, in getattr func = self.getitem(name) ^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\dmsha\miniconda3\Lib\ctypes\init.py", line 394, in getitem__ func = self._FuncPtr((name_or_ordinal, self)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: function 'TVMGetLastPythonError' not found. Did you mean: 'TVMAPISetLastPythonError'?

As before, I expect a folder containing the weights and library files

Environment

python -c "import tvm; print(tvm._ffi.base._LIB)" <CDLL 'C:\Users\dmsha\miniconda3\Lib\site-packages\tvm\tvm.dll', handle 7ffa41c20000 at 0x230eabbea10>

Additional context

A blank folder is C:\Users\dmsha\dev\mlc\dist\Llama-2-7b-chat-hf-q4f16_1 is created

junrushao commented 1 year ago

this is an issue from https://github.com/apache/tvm/pull/15596. seems there have been multiple reports in MLC LLM. CC @Lunderberg the original author of this PR if you could take a look

junrushao commented 1 year ago

Would you mind sharing a Python stacktrace to this error msg?

The following 2 lines reproduces themselves many times:
[14:37:25] D:\a\package\package\tvm\src\node\reflection.cc:109: AttributeError: relax.expr.Var object has no attributed shard_dim
Stack trace not available when DMLC_LOG_STACK_TRACE is disabled at compile time.
[14:37:25] D:\a\package\package\tvm\src\node\reflection.cc:109: AttributeError: relax.expr.Var object has no attributed shard_strategy
Stack trace not available when DMLC_LOG_STACK_TRACE is disabled at compile time.

This will help us find out where this .shard_dim is used exactly in Python codebase

Lunderberg commented 1 year ago

Hmm. The AttributeError: function 'TVMGetLastPythonError' not found. seems rather odd. Did you recompile TVM after pulling?

tqchen commented 1 year ago

I added a note in another thread. By default MSVC do not export the function that are explicitly marked as exported. So in this case it is because TVMGetLastPythonError is not marked via TVM_DLL in declaration header

Lunderberg commented 1 year ago

Ah, Windows seems to be the key point, as Windows doesn't expose symbols by default, and so the extern "C" is insufficient. I had been thinking about exposing to non-tvm libraries, and missed the exposure to other portions of tvm on windows.

Can you try with this hotfix applied?

Edit: Hehe, good timing @tqchen, and I like that it looks like a simple fix. :grin:

tqchen commented 1 year ago

Another fix that likely can resolve this issue of build https://github.com/apache/tvm/pull/15973

junrushao commented 1 year ago

A few related fixes:

Sing-Li commented 1 year ago
image

Please pardon my (possible) related comment. The Note section above. Downloading and "rename it to zstd.dll and copy to the same folder as tvm.dll" part of the instruction --- is almost impossible for new users to accomplish because they have to understand how conda relates to python, and where miniconda keeps its python site libs before they can find "the same folder as tvm.dll". I hope one of the above fixes will make sure that zstd.dll is always included as part of the tvm bundle/nightly 🙏

junrushao commented 1 year ago

That's a nightmare I was trying hard to solve :((

The zstd.dll dependency is introduced by LLVM which we use to generate efficient code, but somehow it is not shipped by default in some Windows distributions...I was trying to static link it into libtvm.dll, but it failed miserably in many different ways...CC @tqchen if you have a better idea :((

tqchen commented 1 year ago

wonder if wheel bunlder can ship that like other ones

David-Sharma commented 1 year ago

Can you solve this by a modification of the docs? (It's how I do understand it)

Original docs: It is likely zstd, a dependency to LLVM, was missing. Please download the precompiled binary, rename it to zstd.dll and copy to the same folder as tvm.dll

Modified: It is likely zstd, a dependency to LLVM, was missing. Please download the precompiled binary, rename it to zstd.dll and copy to the same folder as tvm.dll. Hint - Perform a search for "tvm.dll" and identify the folder in which the path includes the name of the current environment eg. mlc-chat-venv. Copy zstd.dll to that folder.

tqchen commented 1 year ago

@David-Sharma great suggestions, do you mind open a PR

David-Sharma commented 1 year ago

@tqchen Submitted https://github.com/mlc-ai/mlc-llm/issues/1135

tqchen commented 1 year ago

@David-Sharma do you mind directly fork and update the respective files under https://github.com/mlc-ai/mlc-llm/tree/main/docs

junrushao commented 11 months ago

I think It's fixed now :))

junrushao commented 11 months ago

Let's consolidate this to #1135. Let me know if it works now btw!