search

mlc-ai / mlc-llm

Universal LLM Deployment Engine with ML Compilation
https://llm.mlc.ai/
Apache License 2.0
19.07k stars 1.56k forks source link

[Bug] weight conversion error #1738

Closed Vinaysukhesh98 closed 5 months ago

Vinaysukhesh98 commented 8 months ago

🐛 Bug

MODEL_TYPE=minicpm_v (tvm-build-venv)~/Documents/mlc-MiniCPM$ mlc_chat convert_weight --model-type ${MODEL_TYPE} ./dist/models/${MODEL_NAME}/ --quantization $QUANTIZATION -o dist/$MODEL_NAME/ [2024-02-10 15:26:24] INFO auto_config.py:115: Found model configuration: dist/models/MiniCPM-V/config.json [2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: cuda:0 [2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: rocm:0 [2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: metal:0 [2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: vulkan:0 [2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: opencl:0 [2024-02-10 15:26:25] INFO auto_device.py:31: Not found: No available device detected [2024-02-10 15:26:25] INFO auto_weight.py:70: Finding weights in: dist/models/MiniCPM-V [2024-02-10 15:26:25] INFO auto_weight.py:136: Not found Huggingface PyTorch [2024-02-10 15:26:25] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: dist/models/MiniCPM-V/model.safetensors.index.json [2024-02-10 15:26:25] INFO auto_weight.py:106: Using source weight configuration: dist/models/MiniCPM-V/model.safetensors.index.json. Use --source to override. [2024-02-10 15:26:25] INFO auto_weight.py:110: Using source weight format: huggingface-safetensor. Use --source-format to override. [2024-02-10 15:26:25] INFO auto_config.py:153: Found model type: minicpm_v. Use --model-type to override. Traceback (most recent call last): File ".local/bin/mlc_chat", line 33, in sys.exit(load_entry_point('mlc-chat', 'console_scripts', 'mlc_chat')()) File "/Documents/mlc-MiniCPM/python/mlc_chat/main.py", line 28, in main cli.main(sys.argv[2:]) File "/Documents/mlc-MiniCPM/python/mlc_chat/cli/convert_weight.py", line 87, in main convert_weight( File "/Documents/mlc-MiniCPM/python/mlc_chat/interface/convert_weight.py", line 155, in convert_weight args.display() File "Documents/mlc-MiniCPM/python/mlc_chat/interface/convert_weight.py", line 46, in display print(f" {bold('--device'):<25} {_device_to_str(self.device)}", file=out) File "Documents/mlc-MiniCPM/python/mlc_chat/interface/convert_weight.py", line 39, in _device_to_str return f"{Device.MASK2STR[device.device_type]}:{device.device_id}" AttributeError: 'NoneType' object has no attribute 'device_type'

shreygupta2809 commented 8 months ago 
[2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: cuda:0
[2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: rocm:0
[2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: metal:0
[2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: vulkan:0
[2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: opencl:0
[2024-02-10 15:26:25] INFO auto_device.py:31: Not found: No available device detected

It is not able to detect a GPU. Do you have a GPU and support for any other these AUTO_DETECT_DEVICES = ["cuda", "rocm", "metal", "vulkan", "opencl"]? Can you pass --device DEVICE in the above command

Vinaysukhesh98 commented 8 months ago

mlc_chat gen_config --model-type ${MODEL_TYPE} ./dist/models/${MODEL_NAME}/ --quantization $QUANTIZATION --conv-template LM --sliding-window-size 768 -o dist/${MODEL_NAME}/ [2024-02-10 22:18:41] INFO auto_config.py:115: Found model configuration: dist/models/MiniCPM-V/config.json [2024-02-10 22:18:41] INFO auto_config.py:153: Found model type: minicpm_v. Use --model-type to override. [2024-02-10 22:18:41] INFO mistral_model.py:55: prefill_chunk_size defaults to sliding_window_size (4096) [2024-02-10 22:18:41] INFO compiler_flags.py:118: Overriding sliding_window_size from 4096 to 768 [2024-02-10 22:18:41] INFO mistral_model.py:63: Overriding prefill_chunk_size from 4096 to 768 (sliding_window_size) [2024-02-10 22:18:41] INFO gen_config.py:117: [generation_config.json] Setting bos_token_id: 1 [2024-02-10 22:18:41] INFO gen_config.py:117: [generation_config.json] Setting eos_token_id: 2 [2024-02-10 22:18:41] INFO gen_config.py:129: Found tokenizer config: dist/models/MiniCPM-V/tokenizer.model. Copying to dist/MiniCPM-V/tokenizer.model [2024-02-10 22:18:41] INFO gen_config.py:131: Not found tokenizer config: dist/models/MiniCPM-V/tokenizer.json [2024-02-10 22:18:41] INFO gen_config.py:131: Not found tokenizer config: dist/models/MiniCPM-V/vocab.json [2024-02-10 22:18:41] INFO gen_config.py:131: Not found tokenizer config: dist/models/MiniCPM-V/merges.txt [2024-02-10 22:18:41] INFO gen_config.py:131: Not found tokenizer config: dist/models/MiniCPM-V/added_tokens.json [2024-02-10 22:18:41] INFO gen_config.py:129: Found tokenizer config: dist/models/MiniCPM-V/tokenizer_config.json. Copying to dist/MiniCPM-V/tokenizer_config.json [2024-02-10 22:18:41] INFO gen_config.py:137: The model has tokenizer.model but not tokenizer.json. It is always recommended to prefer JSON instead. Attempting to convert using HuggingFace transformers library The repository for dist/models/MiniCPM-V contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/dist/models/MiniCPM-V. You can avoid this prompt in future by passing the argument trust_remote_code=True.

Do you wish to run the custom code? [y/N] n [2024-02-10 22:18:45] ERROR gen_config.py:153: Failed with the exception below. Skipping Traceback (most recent call last): File "/home//Documents/mlc-MiniCPM/python/mlc_chat/interface/gen_config.py", line 148, in gen_config fast_tokenizer = AutoTokenizer.from_pretrained(str(config.parent), use_fast=True) File "/home//.local/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 788, in from_pretrained trust_remote_code = resolve_trust_remote_code( File "/home/.local/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 621, in resolve_trust_remote_code raise ValueError( ValueError: Loading dist/models/MiniCPM-V requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option trust_remote_code=True to remove this error. [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting pad_token_id: 0 [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting temperature: 1.0 [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting repetition_penalty: 1.0 [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting top_p: 0.5 [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting mean_gen_len: 128 [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting max_gen_len: 512 [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting shift_fill_factor: 0.3 [2024-02-10 22:18:45] INFO gen_config.py:159: Dumping configuration file to: dist/MiniCPM-V/mlc-chat-config.json

Vinaysukhesh98 commented 8 months ago 
[2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: cuda:0
[2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: rocm:0
[2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: metal:0
[2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: vulkan:0
[2024-02-10 15:26:25] INFO auto_device.py:85: Not found device: opencl:0
[2024-02-10 15:26:25] INFO auto_device.py:31: Not found: No available device detected

It is not able to detect a GPU. Do you have a GPU and support for any other these AUTO_DETECT_DEVICES = ["cuda", "rocm", "metal", "vulkan", "opencl"]? Can you pass --device DEVICE in the above command

This helps ! note: also we need to compile our pc's any one of the available gpu , cpu won't works., here i have compiled with vulkan and restart the pc to make it work

Vinaysukhesh98 commented 8 months ago

mlc_chat gen_config --model-type ${MODEL_TYPE} ./dist/models/${MODEL_NAME}/ --quantization QUANTIZATION−−conv−templateLM−−sliding−window−size768−odist/{MODEL_NAME}/ [2024-02-10 22:18:41] INFO auto_config.py:115: Found model configuration: dist/models/MiniCPM-V/config.json [2024-02-10 22:18:41] INFO auto_config.py:153: Found model type: minicpm_v. Use --model-type to override. [2024-02-10 22:18:41] INFO mistral_model.py:55: prefill_chunk_size defaults to sliding_window_size (4096) [2024-02-10 22:18:41] INFO compiler_flags.py:118: Overriding sliding_window_size from 4096 to 768 [2024-02-10 22:18:41] INFO mistral_model.py:63: Overriding prefill_chunk_size from 4096 to 768 (sliding_window_size) [2024-02-10 22:18:41] INFO gen_config.py:117: [generation_config.json] Setting bos_token_id: 1 [2024-02-10 22:18:41] INFO gen_config.py:117: [generation_config.json] Setting eos_token_id: 2 [2024-02-10 22:18:41] INFO gen_config.py:129: Found tokenizer config: dist/models/MiniCPM-V/tokenizer.model. Copying to dist/MiniCPM-V/tokenizer.model [2024-02-10 22:18:41] INFO gen_config.py:131: Not found tokenizer config: dist/models/MiniCPM-V/tokenizer.json [2024-02-10 22:18:41] INFO gen_config.py:131: Not found tokenizer config: dist/models/MiniCPM-V/vocab.json [2024-02-10 22:18:41] INFO gen_config.py:131: Not found tokenizer config: dist/models/MiniCPM-V/merges.txt [2024-02-10 22:18:41] INFO gen_config.py:131: Not found tokenizer config: dist/models/MiniCPM-V/added_tokens.json [2024-02-10 22:18:41] INFO gen_config.py:129: Found tokenizer config: dist/models/MiniCPM-V/tokenizer_config.json. Copying to dist/MiniCPM-V/tokenizer_config.json [2024-02-10 22:18:41] INFO gen_config.py:137: The model has tokenizer.model but not tokenizer.json. It is always recommended to prefer JSON instead. Attempting to convert using HuggingFace transformers library The repository for dist/models/MiniCPM-V contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/dist/models/MiniCPM-V. You can avoid this prompt in future by passing the argument trust_remote_code=True.

Do you wish to run the custom code? [y/N] n [2024-02-10 22:18:45] ERROR gen_config.py:153: Failed with the exception below. Skipping Traceback (most recent call last): File "/home/mbuhyd/Documents/mlc-MiniCPM/python/mlc_chat/interface/gen_config.py", line 148, in gen_config fast_tokenizer = AutoTokenizer.from_pretrained(str(config.parent), use_fast=True) File "/home/mbuhyd/.local/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 788, in from_pretrained trust_remote_code = resolve_trust_remote_code( File "/home/mbuhyd/.local/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 621, in resolve_trust_remote_code raise ValueError( ValueError: Loading dist/models/MiniCPM-V requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option trust_remote_code=True to remove this error. [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting pad_token_id: 0 [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting temperature: 1.0 [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting repetition_penalty: 1.0 [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting top_p: 0.5 [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting mean_gen_len: 128 [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting max_gen_len: 512 [2024-02-10 22:18:45] INFO gen_config.py:70: [System default] Setting shift_fill_factor: 0.3 [2024-02-10 22:18:45] INFO gen_config.py:159: Dumping configuration file to: dist/MiniCPM-V/mlc-chat-config.json

How to resolve this

shreygupta2809 commented 8 months ago

Maybe this might help #1727 and #1705

jpf888 commented 7 months ago

@Vinaysukhesh98
hello ，We have also recently tried to deploy MiniCPM-v， I want to know if you successfully deployed minicpm-v in mlc-llm？

Vinaysukhesh98 commented 7 months ago

@Vinaysukhesh98 hello ，We have also recently tried to deploy MiniCPM-v， I want to know if you successfully deployed minicpm-v in mlc-llm？

No

jpf888 commented 7 months ago

@Vinaysukhesh98 hello ，We have also recently tried to deploy MiniCPM-v， I want to know if you successfully deployed minicpm-v in mlc-llm？

No

I see this project supports minicmp-v， What's the main problem with the failure you mentioned? https://github.com/OpenBMB/mlc-MiniCPM/tree/main

Vinaysukhesh98 commented 7 months ago

Traceback (most recent call last): /anaconda3/envs/flows/bin/mlc_chat", line 33, in sys.exit(load_entry_point('mlc-chat', 'console_scripts', 'mlc_chat')()) Documents/mlc-MiniCPM/python/mlc_chat/main.py", line 28, in main cli.main(sys.argv[2:]) File Documents/mlc-MiniCPM/python/mlc_chat/cli/convert_weight.py", line 87, in main convert_weight( File Documents/mlc-MiniCPM/python/mlc_chat/interface/convert_weight.py", line 156, in convert_weight _convert_args(args) File Documents/mlc-MiniCPM/python/mlc_chat/interface/convert_weight.py", line 76, in _convertargs , _namedparams, = model.export_tvm( # type: ignore[misc] Fileanaconda3/envs/flows/lib/python3.10/site-packages/tvm/relax/frontend/nn/core.py", line 489, in export_tvm mod, params, ext_mods = Exporter(debug=debug).build(spec) File anaconda3/envs/flows/lib/python3.10/site-packages/tvm/relax/frontend/nn/exporter.py", line 139, in build assert rx.analysis.well_formed(mod) AssertionError