Regarding results

[vanzytay]

Hi,

I was wondering if you managed to get any results close to the paper with this repository?

I downloaded the code and seem to get way worst results from the paper. Is it because of the unreported hyperparameters?

Thanks!

[jaredwei01]

Hi, Have you got any results close the paper? In my experiment, it got a near 10% lower than the paper.

[vanzytay]

@jacobwei

Nope. With this repo I got about there, 10% lower. Definitely nothing close to the range of 70s. In fact, it's worst than LSTMs.

I am trying to reimplement the model myself from scratch. And it also seems like the results are around 65-ish. Have tried various hyperparameters, including different optimizers, learn rates etc. Still the same.

[ganeshjawahar]

@vanzytay As I've mentioned in the README, I guess the difference in the results is because of the unreported hyper-parameters. But now that the original author has released his code, I've added this task of reaching the state-of-the-art performance to my todo list.

[ganeshjawahar]

As I just glanced through the author's code, I see some serious differences: a) Use Stanford Tokenizer to tokenize the sentence b) Replace the aspect term in the original sentence with a placeholder (say '$T'). c) ...

[jurukode]

Can you share where is the location of original author code?

Thank you

[ganeshjawahar]

http://ir.hit.edu.cn/~dytang/paper/aspect_memnet/src.zip

[jurukode]

Thanks @ganeshjawahar,

unfortunately it is written on Java :(

[pcgreat]

I tried to improve the performance by tweaking the model and parameters, and finally got accuracy of 0.69 for Laptop, and 0.77 for Restaurant, which is still not perfect, but much closer to those ofthe original paper (0.72 for Laptop, 0.809 for Resturant). You can checkout my fix here https://github.com/pcgreat/mem_absa if you are interested.

@ganeshjawahar thanks for your great implementation, especially for the attention part. I also want to take a look at original author code, but the link is invalid now. Would you please share it with me if the license permits?

[verazuo]

@pcgreat The code of the original author can't be downloaded anymore, do you have a copy file of the code? Could you please send it to me ? Thanks.

[nkooli]

@pcgreat In your code i noticed that you feed the target (the sentiment ground truth) to the graph in the test method (model.py), can y explicate me why please ?

[pcgreat]

@nkooli yeah, target is fed into the model to calculate the test loss. But don't worry, the trained model won't be changed by this, as long as the self.optim is not triggered (no back prorogation, no update to the variables). But if you are not assured, you are welcome to move the target out of model and compare the results

[wxybdth]

@ganeshjawahar The code from on http://ir.hit.edu.cn/~dytang/paper/aspect_memnet/src.zip cannot be downloaded，would you please share it with me , if possible? Thanks.

[Aron9080]

@pcgreat I downloaded the code and seem that the result is not perfect(0.7 for restaurant) from https://github.com/pcgreat/mem_absa. You have mentioned in the README, the model can improve the accuracy by 8%, the accuracy of restaurant is around 78%. Do you have any idea for me?