search

microsoft / CLAP

Learning audio concepts from natural language supervision
MIT License
455 stars 35 forks source link

Error loading CLAP checkpoint on GPU cluster: unexpected key(s) in state_dict #35

Closed PabloDumenil closed 3 months ago

PabloDumenil commented 3 months ago

I am encountering a RuntimeError when trying to load a checkpoint using the CLAP model on a GPU cluster. The error message indicates that there are unexpected key(s) in the state_dict. What is strange is that it works perfectly on my machine, but as soon as I run it on the gpu, it doesn't work anymore.

Error Message

RuntimeError Traceback (most recent call last) Cell In[2], line 8 6 # Load the CLAP model 7 model = laion_clap.CLAP_Module(enable_fusion=False) ----> 8 model.load_ckpt()

File /scratch/7447183/conda/envs/Py3108/lib/python3.10/site-packages/laion_clap/hook.py:114, in CLAP_Module.load_ckpt(self, ckpt, model_id) 112 print('Load Checkpoint...') 113 ckpt = load_state_dict(ckpt, skip_params=True) --> 114 self.model.load_state_dict(ckpt) 115 param_names = [n for n, p in self.model.named_parameters()] 116 for n in param_names:

File /scratch/7447183/conda/envs/Py3108/lib/python3.10/site-packages/torch/nn/modules/module.py:2189, in Module.load_state_dict(self, state_dict, strict, assign) 2184 error_msgs.insert( 2185 0, 'Missing key(s) in state_dict: {}. '.format( 2186 ', '.join(f'"{k}"' for k in missing_keys))) 2188 if len(error_msgs) > 0: -> 2189 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( 2190 self.class.name, "\n\t".join(error_msgs))) 2191 return _IncompatibleKeys(missing_keys, unexpected_keys)

RuntimeError: Error(s) in loading state_dict for CLAP: Unexpected key(s) in state_dict: "text_branch.embeddings.position_ids".

Steps to Reproduce

  1. Install laion-clap using pip.
  2. Attempt to load the checkpoint using the following code: 
    
import torch
import laion_clap

Load the CLAP model

model = laion_clap.CLAP_Module(enable_fusion=False) model.load_ckpt()

What I Have Tried

Verified that all dependencies are correctly installed. Attempted to filter the state_dict to remove unexpected keys, but encountered further issues.

bmartin1 commented 3 months ago

It seems you are trying to load LAION CLAP. This GitHub repo is for MSCLAP. Be sure to try both of them :)