search

zhongkaifu / Seq2SeqSharp

Seq2SeqSharp is a tensor based fast & flexible deep neural network framework written by .NET (C#). It has many highlighted features, such as automatic differentiation, different network types (Transformer, LSTM, BiLSTM and so on), multi-GPUs supported, cross-platforms (Windows, Linux, x86, x64, ARM), multimodal model for text and images and so on.
Other
193 stars 38 forks source link

Matrix initialization method #68

Closed calope12 closed 1 year ago

calope12 commented 1 year ago

Hi, Zhongkaifu.

I would want to know what is the initialization method used for the matrices.

Thanks a lot.

zhongkaifu commented 1 year ago

Hi @calope12

You could check initialization method here: https://github.com/zhongkaifu/Seq2SeqSharp/blob/da0469c4afc758543c0edbe0d0be1a28ed0f990e/Seq2SeqSharp/Tools/WeightTensor.cs#L186-L206

Thanks Zhongkai Fu

calope12 commented 1 year ago

Hi, Zhongkaifu.

I have checked the code and I've drawn the conclusion that it is a Xavier initialization. Could you confirm this, please? Just in case I misunderstood it.

Thanks, again

zhongkaifu commented 1 year ago

Yes. You are correct. You could modify or add new initialization in code.

calope12 commented 1 year ago

Thanks again!