How to save the trained weight?

[faruk-ahmad]

We are using kur for training a speech model. But after the training finishes, no weights are stored. However we have tried with: weights: path and also with this format, weights: initial: path save_best: path last: path

but at the end of the training it just create a directory, no weight file is generated. What could be the problem and the solution. Any help would be highly appreciated. TIA

[ajsyp]

Are you sure the weights directory is empty? The individual weight files may have been saved with a leading period ("."), making then hidden files on POSIX systems. When you cd into the weights directory, what does ls -a return?

[faruk-ahmad]

Let me check and get back to you. Thanks. @ajsyp

[faruk-ahmad]

When I do, ls -a in the weight directory I got the following output:

..batch_normalization.0+..batch_normalization.0_beta:0.kur ..batch_normalization.0+..batch_normalization.0_gamma:0.kur ..batch_normalization.0+..batch_normalization.0_moving_mean:0.kur ..batch_normalization.0+..batch_normalization.0_moving_variance:0.kur ..batch_normalization.1+..batch_normalization.1_beta:0.kur ..batch_normalization.1+..batch_normalization.1_gamma:0.kur ..batch_normalization.1+..batch_normalization.1_moving_mean:0.kur ..batch_normalization.1+..batch_normalization.1_moving_variance:0.kur ..batch_normalization.2+..batch_normalization.2_beta:0.kur ..batch_normalization.2+..batch_normalization.2_gamma:0.kur ..batch_normalization.2+..batch_normalization.2_moving_mean:0.kur ..batch_normalization.2+..batch_normalization.2_moving_variance:0.kur ..batch_normalization.3+..batch_normalization.3_beta:0.kur ..batch_normalization.3+..batch_normalization.3_gamma:0.kur ..batch_normalization.3+..batch_normalization.3_moving_mean:0.kur ..batch_normalization.3+..batch_normalization.3_moving_variance:0.kur ..convolution.0+..convolution.0_bias:0.kur ..convolution.0+..convolution.0_kernel:0.kur ..parallel.0_..dense.0_0_0+..parallel.0_..dense.0_0_0_bias:0.kur ..parallel.0_..dense.0_0_0+..parallel.0_..dense.0_0_0_kernel:0.kur ..recurrent.0+..recurrent.0_bias:0.kur ..recurrent.0+..recurrent.0_kernel:0.kur ..recurrent.0+..recurrent.0_recurrent_kernel:0.kur ..recurrent.1+..recurrent.1_bias:0.kur ..recurrent.1+..recurrent.1_kernel:0.kur ..recurrent.1+..recurrent.1_recurrent_kernel:0.kur ..recurrent.2+..recurrent.2_bias:0.kur ..recurrent.2+..recurrent.2_kernel:0.kur ..recurrent.2+..recurrent.2_recurrent_kernel:0.kur

So, are those the trained weights? @ajsyp

[ajsyp]

Yep! That's them.

[faruk-ahmad]

Thanks a lot. You really saved me from getting stuck here. :+1: @ajsyp

[ajsyp]

No problem! Pro-tip: if your layers are named, then the names of the saved weights will be somewhat more meaningful :)

[mahi19]

. .. 0 ..batch_normalization.0+..batch_normalization.0_beta 0 ..batch_normalization.0+..batch_normalization.0_gamma 0 ..batch_normalization.0+..batch_normalization.0_moving_mean 0 ..batch_normalization.0+..batch_normalization.0_moving_variance 0 ..batch_normalization.1+..batch_normalization.1_beta 0 ..batch_normalization.1+..batch_normalization.1_gamma 0 ..batch_normalization.1+..batch_normalization.1_moving_mean 0 ..batch_normalization.1+..batch_normalization.1_moving_variance 0 ..batch_normalization.2+..batch_normalization.2_beta 0 ..batch_normalization.2+..batch_normalization.2_gamma 0 ..batch_normalization.2+..batch_normalization.2_moving_mean 0 ..batch_normalization.2+..batch_normalization.2_moving_variance 0 ..batch_normalization.3+..batch_normalization.3_beta 0 ..batch_normalization.3+..batch_normalization.3_gamma 0 ..batch_normalization.3+..batch_normalization.3_moving_mean 0 ..batch_normalization.3+..batch_normalization.3_moving_variance 0 ..convolution.0+..convolution.0_bias 0 ..convolution.0+..convolution.0_kernel 0 ..parallel.0_..dense.0_0_0+..parallel.0_..dense.0_0_0_bias 0 ..parallel.0_..dense.0_0_0+..parallel.0_..dense.0_0_0_kernel 0 ..recurrent.0+..recurrent.0_bias 0 ..recurrent.0+..recurrent.0_kernel 0 ..recurrent.0+..recurrent.0_recurrent_kernel 0 ..recurrent.1+..recurrent.1_bias 0 ..recurrent.1+..recurrent.1_kernel 0 ..recurrent.1+..recurrent.1_recurrent_kernel 0 ..recurrent.2+..recurrent.2_bias 0 ..recurrent.2+..recurrent.2_kernel 0 ..recurrent.2+..recurrent.2_recurrent_kernel ) 0 bytes 30,297,563,136 bytes free it shows 0 bytes. how to restart training with the best weights?