search

predibase / lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
https://loraexchange.ai
Apache License 2.0
2.18k stars 143 forks source link

Qwen-VL failing with --trust-remote-code #178

Closed pbarker closed 10 months ago

pbarker commented 10 months ago

System Info

ghcr.io/predibase/lorax:latest

Information

Tasks

Reproduction

docker run ghcr.io/predibase/lorax:latest --model-id Qwen/Qwen-VL --trust-remote-code

(this is on runpod)

Expected behavior

I see Qwen is supported, not sure if VL has unsupported changes, but here is are the error logs:

2024-01-11T18:54:17.148358877Z 2024-01-11T18:54:17.147981Z INFO lorax_launcher: convert.py:120 Convert: [2/10] -- Took: 0:00:06.356602 2024-01-11T18:54:17.148409287Z 2024-01-11T18:54:22.044956069Z 2024-01-11T18:54:22.044625Z ERROR download: lorax_launcher: Download encountered an error: Traceback (most recent call last): 2024-01-11T18:54:22.045006167Z 2024-01-11T18:54:22.045008251Z File "/opt/conda/bin/lorax-server", line 8, in 2024-01-11T18:54:22.045010255Z sys.exit(app()) 2024-01-11T18:54:22.045013370Z 2024-01-11T18:54:22.045015022Z File "/opt/conda/lib/python3.10/site-packages/lorax_server/cli.py", line 201, in download_weights 2024-01-11T18:54:22.045018779Z _download_weights(model_id, revision, extension, auto_convert, source) 2024-01-11T18:54:22.045023813Z 2024-01-11T18:54:22.045025582Z File "/opt/conda/lib/python3.10/site-packages/lorax_server/cli.py", line 175, in _download_weights 2024-01-11T18:54:22.045028167Z utils.convert_files(local_pt_files, local_st_files, discard_names) 2024-01-11T18:54:22.045030937Z 2024-01-11T18:54:22.045033744Z File "/opt/conda/lib/python3.10/site-packages/lorax_server/utils/convert.py", line 118, in convert_files 2024-01-11T18:54:22.045035509Z convert_file(pt_file, sf_file, discard_names) 2024-01-11T18:54:22.044999070Z Error: DownloadError 2024-01-11T18:54:22.045037891Z 2024-01-11T18:54:22.045088499Z File "/opt/conda/lib/python3.10/site-packages/lorax_server/utils/convert.py", line 89, in convert_file 2024-01-11T18:54:22.045097622Z save_file(loaded, sf_file, metadata=metadata) 2024-01-11T18:54:22.045102755Z 2024-01-11T18:54:22.045107405Z File "/opt/conda/lib/python3.10/site-packages/safetensors/torch.py", line 232, in save_file 2024-01-11T18:54:22.045112412Z serialize_file(_flatten(tensors), filename, metadata=metadata) 2024-01-11T18:54:22.045118999Z 2024-01-11T18:54:22.045122885Z File "/opt/conda/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 580, in _raise_timeout_error 2024-01-11T18:54:22.045129047Z raise ValueError( 2024-01-11T18:54:22.045133429Z 2024-01-11T18:54:22.045137725Z ValueError: Loading this model requires you to execute custom code contained in the model repository on your local machine. Please set the option trust_remote_code=True to permit loading of this model. 2024-01-11T18:54:22.045159137Z 2024-01-11T18:54:22.045163232Z

tgaddair commented 10 months ago

Hey @pbarker, I believe Qwen-VL is a visual language model. We don't currently support VLMs, though it is something that's on the backlog (#179). The error message, however, is not informative. We should at least improve that.

Do you get the same error with, say, Qwen-7b-chat? https://huggingface.co/Qwen/Qwen-7B-Chat

pbarker commented 10 months ago

Sounds good, let me know if you are looking for contributors in this space.

tgaddair commented 10 months ago

Hey @pbarker, definitely, we'd love to have more contributors. Let me know if you're interested in helping with #179!