tagset beyond Penn Treebank

[Oneplus]

Hi Tao,

Thank you for releasing your code. We are trying it on some new tweet annotation but the code raised the following errors.

Traceback (most recent call last):
  File "adv_train.py", line 638, in <module>
    main()
  File "adv_train.py", line 62, in main
    FLAGS.batch_size, FLAGS.num_classes)
  File "/users1/yjliu/work/project/TPANN/data_helper.py", line 172, in __init__
    label_tensor = to_categorical(label_tensor,num_classes=num_class)
  File "/users1/yjliu/work/project/TPANN/np_utils.py", line 13, in to_categorical
    categorical[np.arange(n), y] = 1
IndexError: index 53 is out of bounds for axis 1 with size 53

We believe it's because we are testing the model on a new tag set beyond PTB. Do you have any suggestion on how to make it work on POS data which were annotated following different tag set.

Regards,

[Oneplus]

To be more specific, we are working on Universal POS tag set.

It's feasible for us to convert PTB tag to universal POS tag. But we believe the PTB data is released through the binary file in your code, which make it impossible for us to switch to PTB with universal POS tag. Do you have any suggestion for us to hack into the data loading process (data_helper.load_data).

Thanks.

[xiutangzju]

Hi, I have met some problems about retraining the model. I want to know did you retrain the model successfully with the data provided by the author.

I had successfully reproduced the results of paper by using python adv_train.py --choice 0 , but when i try to retrain the model by using $ python adv_train.py --choice 1 . The result shows that the model loss is nan, and the accuracy rate is zero, resulting in the failure to store the model. The print what I got is as the following:

the 200 time of PTB class  loss is:nan
the 200 time of PTB class accuracy is:0.000000
the 200 time of PTB domain loss is:0.692448
the 200 time of PTB domain accuracy is:0.395349
the 200 time of TWE class loss is:nan
the 200 time of TWE class accuracy is:1.000000
the 200 time of TWE domain loss is:0.700776
the 200 time of TWE autoencoder_cost is:nan
the 200 time of TWE domain accuracy is:0.344371

I don't know what is the cause of the problem. Have you ever met this problem? I hope you can give me a suggestion. Thank you very much.

[guitaowufeng]

Sorry for the late reply. I didn't notice the issues you posted. I think you may introduce extra labels that may increase the size of the label set. We convert PTB tag to universal POS tag according to the PTB POS Tagging Guidelines (Santorini, 1990) and ARK Guidelines (http://www.ark.cs.cmu.edu/TweetNLP/).