Adding HAST as a supervised baseline

[farinamhz]

We are going to add HAST as a supervised baseline to our experiment for data augmentation in aspect detection task. It has been published in IJCAI 2018 as "Aspect Term Extraction with History Attention and Selective Transformation".

[farinamhz]

So far, we have understood that for running the HAST code line, we need a wrapper for changing our format of the dataset to the one which is suitable for HAST as it has not been provided how to change the original XML version of semeval datasets to the suitable one of HAST. (suitable format of HAST is "S####x0=O, x1=O, x2=T, x3=O" in which S is the full review sentence and x0, x1, x2, ... are the words, and also T means aspect or target, and other words will be O.

[farinamhz]

(To be updated)

• Also, there is another file for the opinion (words and sentiment) that we make similar for all as we do not have the annotations from original XML datasets of semeval, and also, our task is aspect extraction, not aspect and opinion together.
• It is better to run the codeline in Linux instead of Windows due to problems with installing dynet library in Windows, so we chose to use Colab instead.
• tabulate library is also part of the requirements which should be installed and has not been mentioned in the README file.
• You need to change the number of reviews in the training dataset, which is static in the code and does not get it from the input arguments.
• The path of embeddings in the main.py file should be changed, and you need to download them from their website into a directory and put the directory's path in the main.py file.
• There is no directory named log in the HAST main directory, so you need to make it yourself.
• Then, you will be able to run the HAST codeline and logs will be stored in the log directory that you have made already in the main directory.

[farinamhz]

The wrapper has been completed for the original dataset and all versions of datasets have been created based on the train as train and valid as test for different percentages of hidden aspects.

The only problem was that the datasets of HAST had special characters, which is #### for separating the tags and sentences, so I changed the characters of hidden aspect from #### to ***** for this baseline.

I have uploaded the code with this special dataset: semeval-14-rest (train and test with 100% hidden aspect) to test the HAST code for our dataset on computecanada.

As soon as I get the results, I will post the update here and start the testing on all versions of semeval with different percentages of hidden aspects.

[farinamhz]

Hi @hosseinfani, I checked the HAST code and results as well as the architecture described in the paper and found out that for each sentence or actually each review, it works like this:

[hosseinfani]

@farinamhz thank you.

• we can simply include a list of words that is selected as T and pair it with the prob and order them. In you example, it becomes [(second word, 0.4845), (tenth word, 0.0504)]. But then, the top-k for k > 2 will be empty, which is ok with pytrec_eval
• or, we can include all words of the review sentence, ordered by the third column prob (ignoring the O tag)

What do you think?

[farinamhz]

Both suggestions seem reasonable, and I suggest doing both and seeing the results of pytrec. I can add a func to the evaluation of HAST and share the results here to decide between these two options. @hosseinfani

[farinamhz]

I have some questions @hosseinfani

• These are not 0<prob<1 as they have values more than 1.
• Also, we have two numbers for each as the second and third values are both target (aspect), and only the first one is o which is outside. In fact, they will be O, B, and I meaning outside, beginning, and Inside, in which beginning and Inside are both considered as T for the target (aspect).
• So which prob should be selected to be stored and ordered?

Do you have any idea about these?

[hosseinfani]

@farinamhz

• not important. we need the order
• ok. for each word, we select the max value (either B or I), then we order them

[farinamhz]

Hi @hosseinfani,

Update on the evaluation of HAST:

This is an example file of the results in HAST that I have changed the evaluation to one similar to our pipeline. If it is ok, I will move on to the new baselines.

pred.eval.mean.csv

• Success @k example:

[hosseinfani]

@farinamhz perfect. btw, we only need 1,5,10,100