size mismatch RuntimeError running CLI demo

[krisgesling]

Hey there, I was just trying to recreate the Hey Firefox model using the provided datasets. Training seems to go smoothly using PR #32 but when attempting to run the CLI demo with

python -m howl.run.demo --model res8 --workspace workspaces/hey-ff-res8

I'm getting the following RuntimeError:

Traceback (most recent call last):
  File "/home/user/anaconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/user/anaconda3/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/user/howl/howl/run/demo.py", line 106, in <module>
    main()
  File "/home/user/howl/howl/run/demo.py", line 85, in main
    ws.load_model(model, best=True)
  File "/home/user/howl/howl/model/workspace.py", line 47, in load_model
    model.load_state_dict(torch.load(self.model_path(best=best), lambda s, l: s))
  File "/home/user/howl/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 846, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for Res8:
    size mismatch for output.weight: copying a param with shape torch.Size([4, 45]) from checkpoint, the shape in current model is torch.Size([2, 45]).
    size mismatch for output.bias: copying a param with shape torch.Size([4]) from checkpoint, the shape in current model is torch.Size([2]).

The CLI demo ran on the first model I trained (with custom data) however I couldn't get it to recognize anything so am trying to verify my setup with the Hey Firefox example first.

Any ideas would be greatly appreciated.

Thanks

[ljj7975]

it looks to me that you gave different vocab settings. output size is 4 because there are hey, fire, fox, negative. ur trained model seems to be trained for two keywords

can you try to give proper VOCAB env var?

[krisgesling]

Thanks Brandon,

That makes sense about the output size, however I did run it with that hey firefox vocab list from the README.

The first model I trained that did run the demo CLI was a single keyword + negative. So to ensure nothing from that training was influencing this model I setup a fresh virtual environment and re-downloaded the datasets for a completely new run.

Steps taken:

# Setup environment
git clone https://github.com/castorini/howl && cd howl
sudo apt install python3-pyaudio
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Get datasets
wget http://nlp.rocks/firefox
unzip firefox
wget http://nlp.rocks/ffnoise
unzip ffnoise

# Source default res8 env vars
source envs/res8.env

# Train the model
NOISE_DATASET_PATH=/home/user/howl/MS-SNSD DATASET_PATH=/home/user/howl/speaker-id-split-medium LR_DECAY=0.98 VOCAB='[" hey","fire","fox"]' USE_NOISE_DATASET=True BATCH_SIZE=16 INFERENCE_THRESHOLD=0 NUM_EPOCHS=300 NUM_MELS=40 INFERENCE_SEQUENCE=[0,1,2] MAX_WINDOW_SIZE_SECONDS=0.5 python -m howl.run.train --model res8 --workspace workspaces/hey-ff

This returned an import error as already reported in another issue:

ImportError: /home/user/howl/.venv/lib/python3.8/site-packages/numba/_dynfunc.cpython-38-x86_64-linux-gnu.so: undefined symbol: _PyObject_GC_UNTRACK

So installed numba v0.48.0 and re-ran the training successfully.

Then finally tried running the demo CLI:

python -m howl.run.demo --model res8 --workspace workspaces/hey-ff

and received the same size mismatch error.

Not sure when I'll get time to dig into this more deeply so thought I'd share my process anyway.

Thanks for all the work you've put into this too :100:

[ljj7975]

I will take a look into it later when I have free time. right now I don't have much.

but I will promise you I will get back to this issue