search

zhongkaifu / RNNSharp

RNNSharp is a toolkit of deep recurrent neural network which is widely used for many different kinds of tasks, such as sequence labeling, sequence-to-sequence and so on. It's written by C# language and based on .NET framework 4.6 or above versions. RNNSharp supports many different types of networks, such as forward and bi-directional network, sequence-to-sequence network, and different types of layers, such as LSTM, Softmax, sampled Softmax and others.
BSD 3-Clause "New" or "Revised" License
285 stars 92 forks source link

Saving better model crash. #7

Closed bill3code closed 8 years ago

bill3code commented 8 years ago

Hello.

info,20.02.2016 21:08:27 [TRACE] In training: log probability = -6539,9872918264 9, cross-entropy = 0,0073893168866162, perplexity = 1,00513502343797 info,20.02.2016 21:08:27 info,20.02.2016 21:08:27 Saving better model into file model.bin... info,20.02.2016 21:08:27 Saving input2hidden weights... info,20.02.2016 21:08:28 Saving matrix with VQ 256... info,20.02.2016 21:10:28 Sorting data set (size: 1707219900)...

Process is terminated due to StackOverflowException.

Process is terminated due to StackOverflowException. Process is terminated due to StackOverflowException.

rnn error

Seems an error occurs in AdvUtils.dll

bill3code commented 8 years ago

I'm not sure, but the problem seems to be in the method Split Cluster.

zhongkaifu commented 8 years ago

Hi,

Yes, this is a bug in AdvUtils.dll. The values of data may be not balanced, so we call SplitCluster so deep, and then stack is overflow.

I have checked-in some new RNNSharp code, and disable model vector quantization by default, so far, this problem has been mitigated, and I will fix this problem in AdvUtils project.

I suggest you to use the latest RNNSharp or its demo package, since I have done lots of performance optimization for encoding, especially for LSTM. Your training will be much faster than before.

bill3code commented 8 years ago

Great! Thanks for the support.