search

brightmart / text_classification

all kinds of text classification models and more with deep learning
MIT License
7.83k stars 2.57k forks source link

han-transformer not work #91

Closed Continue7777 closed 5 years ago

Continue7777 commented 5 years ago

i have tried p1_HierarchicalAttention_model_transformer.py with your test function,but the loss dose not converge。 image

brightmart commented 5 years ago

I think you are run on toy task, and loss is decreasing. if you want coverage, you can use real task,

Continue7777 commented 5 years ago

I run p1_HierarchicalAttention_model.py ,its work fine .and i just want to try transformer,but i dosen't work,so i try you test function,but it dose not work too?

brightmart commented 5 years ago

got it. transformer need to fix a problem. before that you can use this:

https://github.com/google-research/bert/blob/master/modeling.py

Continue7777 commented 5 years ago

thanks

brightmart commented 5 years ago

I think transformer work better now. as someone find a bug which was fixed: https://github.com/brightmart/text_classification/issues/93

Continue7777 commented 5 years ago

hey,that's me . and i just try your transofrmer code.it's work. but i campare to textcnn, just as the same conclusion in your github(https://github.com/brightmart/text_classification). transformer isn't better than textcnn ,may be my dataset is not large enough. But, i think transformer may be more potential. I will try pre-train like bert and find whether it's help.

brightmart commented 5 years ago

good to hear from you. please keep me update.