search

vietnh1009 / Hierarchical-attention-networks-pytorch

Hierarchical Attention Networks for document classification
385 stars 104 forks source link

No convergence during fine tuning #7

Open zysNLP opened 5 years ago

zysNLP commented 5 years ago

When I used yahoo pretrained model to fineture my data. Find a misconvergence result below:

Model's parameters: {'batch_size': 128, 'num_epoches': 1, 'lr': 0.01, 'momentum': 0.9, 'word_hidden_size': 50, 'sent_hidden_size': 50, 'es_min_delta': 0.0, 'es_patience': 5, 'pre_trained_model': 'trained_models/yahoo_answers_csv/whole_model_han', 'train_set': 'data/ours/data_train_csv.csv', 'test_set': 'data/ours/data_test_csv.csv', 'test_interval': 1, 'word2vec_path': 'data/glove.6B/glove.6B.300d.txt', 'log_path': 'tensorboard/han_voc', 'saved_path': 'trained_models_ours'}Epoch: 1/1 Test loss: 1.2496397495269775 Test accuracy: 0.44883720930232557 Test confusion matrix: [[ 0 0 0 0 0 1 0 0 0] [ 0 0 0 0 0 4 0 0 0] [ 0 0 0 0 0 8 0 0 0] [ 0 0 0 0 0 88 0 0 0] [ 0 0 0 0 0 831 0 0 0] [ 0 0 0 0 0 1737 0 0 0] [ 0 0 0 0 0 1131 0 0 0] [ 0 0 0 0 0 62 0 0 0] [ 0 0 0 0 0 8 0 0 0]]

Forget num_epoches=1 since more epoches didn't work so I just trained 1. My dataset has similar contents and nums_classes=9 which aslo similar to yahoo 10 so I didn't change this. Question is, why get these results above. Is it really nums_classes‘ problem？

zysNLP commented 5 years ago

Found data have unbalance, fixed it and change num_classes but still have same problem.

RakeJohnTytor commented 3 years ago

Excuse me, could you please share the pretrained models?

Fino2020 commented 2 years ago

I got the same question, could you please tell me whether have you solved this problem?

Fino2020 commented 2 years ago

I trained the model 7times，and it got different loss value，I think maybe the Initialisation get false. and the data in dataset should'n contains nan or null