AssertionError when loading model trained with neon<2.3.0

[Drea1989]

hello, i have tried to load my model trained with neon 2.0 as well as the pretrained model available in the modelzoo returning the same error, only if i train a new model in neon 2.3.0 i am also able to load it. any conversion tool planned?

python evaluate.py --manifest val:/home/drea/odie_cloud/deepspeech/librispeech/dev-clean/val-manifest.tsv --model_file /home/drea/odie_cloud/deepspeech/librispeech_16_epochs.prm
2017-11-07 12:09:37,552 - neon.backends - WARNING - deterministic_update and deterministic args are deprecated in favor of specifying random seed
Traceback (most recent call last):
  File "evaluate.py", line 69, in <module>
    model = Model(args.model_file)
  File "/home/drea/anaconda3/envs/odiestt/lib/python3.5/site-packages/neon-2.3.0-py3.5.egg/neon/models/model.py", line 72, in __init__
    self.load_params(layers, load_states=(not weights_only))
  File "/home/drea/anaconda3/envs/odiestt/lib/python3.5/site-packages/neon-2.3.0-py3.5.egg/neon/models/model.py", line 414, in load_params
    self.deserialize(load_obj(param_path), load_states=load_states)
  File "/home/drea/anaconda3/envs/odiestt/lib/python3.5/site-packages/neon-2.3.0-py3.5.egg/neon/models/model.py", line 475, in deserialize
    self.layers.load_weights(model_dict['model'], load_states)
  File "/home/drea/anaconda3/envs/odiestt/lib/python3.5/site-packages/neon-2.3.0-py3.5.egg/neon/layers/container.py", line 200, in load_weights
    assert len(pdict['config']['layers']) == len(self.layers)
AssertionError

[wei-v-wang]

I use the following "pseudo" code to convert old weights to new weights.

layers = [....] # Conversion routine would require the same layers used to create the old model
model = Model(layers=layers)
model.load_params(old_weight_file, load_state=False/True) load_state=False if no states information are in the old_weight_file, True otherwise model.save_params(new_weight_file)

[wei-v-wang]

Take VGG for example:

[omitting unrelated code ...] Full files can be found at https://github.com/NervanaSystems/ModelZoo/blob/VGG_neon2.3/ImageClassification/ILSVRC2012/VGG/vgg_neon_train.py

54 # Set up the model layers 55 layers = [] 56 57 # set up 3x3 conv stacks with different feature map sizes 58 for nofm in [64, 128, 256, 512, 512]: 59 layers.append(Conv((3, 3, nofm), conv_params)) 60 layers.append(Conv((3, 3, nofm), conv_params)) 61 if nofm > 128: 62 layers.append(Conv((3, 3, nofm), conv_params)) 63 if args.vgg_version == 'E': 64 layers.append(Conv((3, 3, nofm), conv_params)) 65 layers.append(Pooling(2, strides=2)) 66 67 layers.append(Affine(nout=4096, init=initfc, bias=Constant(0), activation=relu)) 68 layers.append(Dropout(keep=0.5)) 69 layers.append(Affine(nout=4096, init=initfc, bias=Constant(0), activation=relu)) 70 layers.append(Dropout(keep=0.5)) 71 layers.append(Affine(nout=1000, init=initfc, bias=Constant(0), activation=Softmax())) 72 73 cost = GeneralizedCost(costfunc=CrossEntropyMulti()) 74 75 model = Model(layers=layers)

Suppose you have old weights named VGG_D.p, assuming the VGG_D.p model contains the same number of layers as layers variable above, you can print out the VGG_D.p to double check. If VGG_D.p does not contain all the above layers (e.g. it does not contain line 66-71 layers), then please delete line 66071 from the layers.

step 1: load it model.load_params(VGG_D.p) Run the program, if VGG_D.p does not contain states information, it will error out and ask you to change the above to: model.load_params(VGG_D.p, load_state=False)

step 2: save it (in new weight format)

model.save_params(VGG_D_fused_conv_bias.p)

Done.

Please let me know if you are able to convert. If you prefer, you can send me the weight file and the script used to generate the weight file, I would be able to convert it for you. Thank you!

[wei-v-wang]

Oh, I see. It is DS2 pre-trained model https://s3-us-west-1.amazonaws.com/nervana-modelzoo/Deep_Speech/Librispeech/librispeech_16_epochs.prm Will work on converting that.

[Drea1989]

hello, thanks for the help, i was able to load and save the new model by printing out the model architecture and verifying that the structure of the layers where equal. all good now i was able to train and infer.

[wei-v-wang]

OK, thanks for the update and I am glad that you have got the new model weights.

For me I had to do the following extra in neon/layers/recurrent.py to convert ds2 model:

            if getattr(self, key) is not None:
                serial_dict['params'][key] = getattr(self, key).get()
            else:
                serial_dict['params'][key] = None

[wei-v-wang]

Closing this issue. Feel free to open new issues.