gradio_app.py

[wwfcnu]

run script ：python gradio_app.py --model ./ckpts/checkpoint.pth --llama_dir ./ckpts/LLaMA --mert_path ./ckpts/MERT-v1-330M

Error log：

/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/site-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in $
.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional.
  warnings.warn(
/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/site-packages/torchvision/transforms/_functional_video.py:6: UserWarning: The 'torchvision.transforms._functional_video' module is deprecated si
nce 0.12 and will be removed in the future. Please use the 'torchvision.transforms.functional' module instead.
  warnings.warn(
/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/site-packages/torchvision/transforms/_transforms_video.py:22: UserWarning: The 'torchvision.transforms._transforms_video' module is deprecated s
ince 0.12 and will be removed in the future. Please use the 'torchvision.transforms' module instead.
  warnings.warn(
Loading LLaMA-Adapter from ./ckpts/checkpoint.pth
WARNING: feature_extractor_cqt requires the libray 'nnAudio'
model args: ModelArgs(dim=4096, n_layers=32, n_heads=32, n_kv_heads=None, vocab_size=-1, multiple_of=256, ffn_dim_multiplier=None, norm_eps=1e-05, max_batch_size=1, max_seq_len=8192, w_bias=True, 
w_lora=True, lora_rank=16)

Traceback (most recent call last):
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/http/client.py", line 1447, in connect
    super().connect()
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/http/client.py", line 946, in connect
    self.sock = self._create_connection(
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/socket.py", line 844, in create_connection
    raise err
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/socket.py", line 832, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py", line 33, in <module>
    model = llama.load(args.model, args.llama_dir, mert_path=args.mert_path, knn=True, llama_type=args.llama_type)
  File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/llama/llama_adapter.py", line 399, in load
    model = LLaMA_adapter(
  File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/llama/llama_adapter.py", line 145, in __init__
    self.index = faiss.read_index(download("https://huggingface.co/csuhan/knn/resolve/main/knn.index", knn_dir))
  File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/util/misc.py", line 404, in download
    with urllib.request.urlopen(url) as source, open(download_target, "wb") as output:
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/home/wangweifei/.conda/envs/mu-llama/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>

[wwfcnu]

I have downloaded the model file locally, why do I still get a network error?

[shansongliu]

File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/llama/llama_adapter.py", line 145, in __init__
   self.index = faiss.read_index(download("https://huggingface.co/csuhan/knn/resolve/main/knn.index", knn_dir))

Have you downloaded the knn.index file? We should have indicated all the model file paths in the code's comment.

[wwfcnu]

Ok, then after I download it locally, how do I specify the path to the knn file? self.index = faiss.read_index("./ckpt/knn.index")?

At 2023-09-26 10:52:55, "shansongliu" @.***> wrote:

File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/llama/llama_adapter.py", line 145, in init self.index = faiss.read_index(download("https://huggingface.co/csuhan/knn/resolve/main/knn.index", knn_dir))

Have you downloaded the knn.index file? We should have indicated all the model file paths in the code's comment.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[shansongliu]

Ok, then after I download it locally, how do I specify the path to the knn file? self.index = faiss.read_index("./ckpt/knn.index")? At 2023-09-26 10:52:55, "shansongliu" @.> wrote: File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/llama/llama_adapter.py", line 145, in init self.index = faiss.read_index(download("https://huggingface.co/csuhan/knn/resolve/main/knn.index", knn_dir)) Have you downloaded the knn.index file? We should have indicated all the model file paths in the code's comment. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.>

As you can see the line 25 of this script "https://github.com/crypto-code/MU-LLaMA/blob/LLaMA-2/MU-LLaMA/llama/llama_adapter.py", the default dir of knn index file is './ckpts', you can try to put your downloaded knn index file there. @crypto-code Help me check whether I am correct or not here. Or shall we also provide an explicit input parameter for the knn index file?

[tanggang1997]

Ok, then after I download it locally, how do I specify the path to the knn file? self.index = faiss.readindex("./ckpt/knn.index")? At 2023-09-26 10:52:55, "shansongliu" @.> wrote: File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/llama/llama_adapter.py", line 145, in init self.index = faiss.read_index(download("https://huggingface.co/csuhan/knn/resolve/main/knn.index", knndir)) Have you downloaded the knn.index file? We should have indicated all the model file paths in the code's comment. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @_._>

As you can see the line 25 of this script "https://github.com/crypto-code/MU-LLaMA/blob/LLaMA-2/MU-LLaMA/llama/llama_adapter.py", the default dir of knn index file is './ckpts', you can try to put your downloaded knn index file there. @crypto-code Help me check whether I am correct or not here. Or shall we also provide an explicit input parameter for the knn index file?

Hi, I'm downing knn model, but now it reports this error rror(s) in loading state_dict for LLaMA_adapter: size mismatch for prefix_query.weight: copying a param with shape torch.Size([32, 4096]) from checkpoint, the shape in current model is torch.Size([20, 4096]). File "/data1/tanggang/MU-LLaMA-LLaMA-2/MU-LLaMA/llama/llama_adapter.py", line 402, in load load_result = model.load_state_dict(adapter_ckpt['model'], strict=False) File "/data1/tanggang/MU-LLaMA-LLaMA-2/MU-LLaMA/gradio_app.py", line 33, in model = llama.load(args.model, args.llama_dir, mert_path=args.mert_path, knn=True, llama_type=args.llama_type) RuntimeError: Error(s) in loading state_dict for LLaMA_adapter: size mismatch for prefix_query.weight: copying a param with shape torch.Size([32, 4096]) from checkpoint, the shape in current model is torch.Size([20, 4096]).

[tanggang1997]

now it has new problem, cause by assert len(load_result.unexpected_keys) == 0, f"Unexpected keys: {load_result.unexpected_keys}"

Unexpected keys: ['image_bind_proj.weight', 'image_bind_proj.bias', 'image_bind_norm_1.weight', 'image_bind_f1_1.weight', 'image_bind_f2_1.weight', 'image_bind_f3_1.weight', 'image_bind_norm_2.weight', 'image_bind_f1_2.weight', 'image_bind_f2_2.weight', 'image_bind_f3_2.weight', 'image_bind_norm_3.weight', 'image_bind_f1_3.weight', 'image_bind_f2_3.weight', 'image_bind_f3_3.weight'] File "/data1/tanggang/MU-LLaMA-LLaMA-2/MU-LLaMA/llama/llama_adapter.py", line 404, in load assert len(load_result.unexpected_keys) == 0, f"Unexpected keys: {load_result.unexpected_keys}" File "/data1/tanggang/MU-LLaMA-LLaMA-2/MU-LLaMA/gradio_app.py", line 33, in model = llama.load(args.model, args.llama_dir, mert_path=args.mert_path, knn=True, llama_type=args.llama_type) AssertionError: Unexpected keys: ['image_bind_proj.weight', 'image_bind_proj.bias', 'image_bind_norm_1.weight', 'image_bind_f1_1.weight', 'image_bind_f2_1.weight', 'image_bind_f3_1.weight', 'image_bind_norm_2.weight', 'image_bind_f1_2.weight', 'image_bind_f2_2.weight', 'image_bind_f3_2.weight', 'image_bind_norm_3.weight', 'image_bind_f1_3.weight', 'image_bind_f2_3.weight', 'image_bind_f3_3.weight']

[crypto-code]

Ok, then after I download it locally, how do I specify the path to the knn file? self.index = faiss.readindex("./ckpt/knn.index")? At 2023-09-26 10:52:55, "shansongliu" @.> wrote: File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/llama/llama_adapter.py", line 145, in init self.index = faiss.read_index(download("https://huggingface.co/csuhan/knn/resolve/main/knn.index", knndir)) Have you downloaded the knn.index file? We should have indicated all the model file paths in the code's comment. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @_._>

As you can see the line 25 of this script "https://github.com/crypto-code/MU-LLaMA/blob/LLaMA-2/MU-LLaMA/llama/llama_adapter.py", the default dir of knn index file is './ckpts', you can try to put your downloaded knn index file there. @crypto-code Help me check whether I am correct or not here. Or shall we also provide an explicit input parameter for the knn index file?

@shansongliu, You are right and it is a good idea to provide an explicit argument for specifying the directory to the knn index file. I have added the argument for KNN directory in both inference.py and gradio_app.py (132f51ed7943d713bf27a18d50da1572ae3fea7f).

usage: gradio_app.py [-h] [--model MODEL] [--llama_type LLAMA_TYPE] [--llama_dir LLAMA_DIR] [--mert_path MERT_PATH] [--knn_dir KNN_DIR]

optional arguments:
  -h, --help            show this help message and exit
  --model MODEL         Name of or path to MU-LLaMA pretrained checkpoint
  --llama_type LLAMA_TYPE
                        Type of llama original weight
  --llama_dir LLAMA_DIR
                        Path to LLaMA pretrained checkpoint
  --mert_path MERT_PATH
                        Path to MERT pretrained checkpoint
  --knn_dir KNN_DIR     Path to directory with KNN Index

The default value for the knn_dir argument is ./ckpts. But you can set it to any directory where you have downloaded the knn.index file.

now it has new problem, cause by assert len(load_result.unexpected_keys) == 0, f"Unexpected keys: {load_result.unexpected_keys}"

Unexpected keys: ['image_bind_proj.weight', 'image_bind_proj.bias', 'image_bind_norm_1.weight', 'image_bind_f1_1.weight', 'image_bind_f2_1.weight', 'image_bind_f3_1.weight', 'image_bind_norm_2.weight', 'image_bind_f1_2.weight', 'image_bind_f2_2.weight', 'image_bind_f3_2.weight', 'image_bind_norm_3.weight', 'image_bind_f1_3.weight', 'image_bind_f2_3.weight', 'image_bind_f3_3.weight'] File "/data1/tanggang/MU-LLaMA-LLaMA-2/MU-LLaMA/llama/llama_adapter.py", line 404, in load assert len(load_result.unexpected_keys) == 0, f"Unexpected keys: {load_result.unexpected_keys}" File "/data1/tanggang/MU-LLaMA-LLaMA-2/MU-LLaMA/gradio_app.py", line 33, in model = llama.load(args.model, args.llama_dir, mert_path=args.mert_path, knn=True, llama_type=args.llama_type) AssertionError: Unexpected keys: ['image_bind_proj.weight', 'image_bind_proj.bias', 'image_bind_norm_1.weight', 'image_bind_f1_1.weight', 'image_bind_f2_1.weight', 'image_bind_f3_1.weight', 'image_bind_norm_2.weight', 'image_bind_f1_2.weight', 'image_bind_f2_2.weight', 'image_bind_f3_2.weight', 'image_bind_norm_3.weight', 'image_bind_f1_3.weight', 'image_bind_f2_3.weight', 'image_bind_f3_3.weight']

This error is because the default argument for model in the gradio_app.py is not set to the model checkpoint file, I have changed it in both gradio_app.py and inference.py. The model argument must point to ./ckpts/checkpoint.pth.

I hope this addresses all your issues and you can successfully run the demo. You can close the issue if this is resolved. We would appreciate you starring our repo 😊.

[wwfcnu]

@crypto-code Traceback (most recent call last): File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py", line 173, in create_imagebind_llm_demo() File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py", line 103, in create_imagebind_llm_demo clear = gr.ClearButton([msg, chatbot]) AttributeError: module 'gradio' has no attribute 'ClearButton'

[tanggang1997]

@crypto-code Traceback (most recent call last): File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py", line 173, in create_imagebind_llm_demo() File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py", line 103, in create_imagebind_llm_demo clear = gr.ClearButton([msg, chatbot]) AttributeError: module 'gradio' has no attribute 'ClearButton'

yes,now I also have this bug

[tanggang1997]

@crypto-code Traceback (most recent call last): File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py", line 173, in create_imagebind_llm_demo() File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py", line 103, in create_imagebind_llm_demo clear = gr.ClearButton([msg, chatbot]) AttributeError: module 'gradio' has no attribute 'ClearButton'

I have already solved this problem, just upgrade the gradio to the latest version

[wwfcnu]

@crypto-code Traceback (most recent call last): File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py", line 173, in create_imagebind_llm_demo() File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py", line 103, in create_imagebind_llm_demo clear = gr.ClearButton([msg, chatbot]) AttributeError: module 'gradio' has no attribute 'ClearButton'回溯（最近一次调用）：文件“/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py”，第 173 行，在 create_imagebind_llm_demo（） 文件中，（） 文件 “/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py”，第 103 行，create_imagebind_llm_demo clear = gr。ClearButton（[msg， chatbot]） AttributeError： 模块 'gradio' 没有属性 'ClearButton'

I have already solved this problem, just upgrade the gradio to the latest version我已经解决了这个问题，只需将gradio升级到最新版本即可

ok，thanks

[shansongliu]

@crypto-code Traceback (most recent call last): File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py", line 173, in create_imagebind_llm_demo() File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py", line 103, in create_imagebind_llm_demo clear = gr.ClearButton([msg, chatbot]) AttributeError: module 'gradio' has no attribute 'ClearButton'

I have already solved this problem, just upgrade the gradio to the latest version

Glad to hear that you solved this problem. Feel free to post any further question or issue when you found a bug of our code. We are willing to help and appreciate that you help us improve our repo😊.

[tanggang1997]

@crypto-code Traceback (most recent call last): File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py", line 173, in create_imagebind_llm_demo() File "/home/wangweifei/repository/wair/MU-LLaMA/MU-LLaMA/gradio_app.py", line 103, in create_imagebind_llm_demo clear = gr.ClearButton([msg, chatbot]) AttributeError: module 'gradio' has no attribute 'ClearButton'

I have already solved this problem, just upgrade the gradio to the latest version

Glad to hear that you solved this problem. Feel free to post any further question or issue when you found a bug of our code. We are willing to help and appreciate that you help us improve our repo😊.

Thank you very much for your work and look forward to your latest research on AI music in the future