"ca" Coherence Metric

[ghost]

Dear Michael Röder,

thanks for sharing this tool.

I played around with different metrics and also took a look at the original papers. Regarding the ca metric, the paper does not really make sense to me.

First the context vector is constructed based on the top n words and their npmi score. The npmi score is between -1 and 1. Now the part that is in my opinion not correct: weighting the elements with gamma (which is 2). This leads to a vector with values between 0 and 1.

For example, assume two vectors, (1,1) and (-1,-1), before weighting. Their cos similarity score is -1, after weighting the cos similarity score becomes 1 .

Is my understanding of the metric correct, or am I missing something?

Thanks in advance Constantin Ponfick

[MichaelRoeder]

Dear Constantin Ponfick,

First, I would like to point out, that the C_A metric has not been defined by us. It has been published by Aletras & Stevenson in 2013 in "Evaluating Topic Coherence Using Distributional Semantics". In Section 3.2.1, the authors explain that they tried different values for γ and they add "For all of our experiments we set γ = 2 which was found to produce the best results."

Your understanding is partly correct. An even gamma value will lead to the effect that negative (N)PMI values become positive. However, your example does not fit very well since a) the cosine similarity is in the range [0,1] and b) the cosine of both original vectors is already 1.0 before applying gamma.

I hope I was able to help you.

Have a nice weekend :smiley:

[ghost]

Thanks for the fast reply. Yes, I know that you are not the original authors. Nevertheless I thought I might just post this here, since you are active in this repo and also used the metric in your experiments.

Could it be, that we both do not use the same definition for the cosine similarity? Per definition (From Frequency to Meaning: Vector Space Models of Semantics) it becomes -1 if two vectors show in exactly opposite direction (my example) and 1 if they show in the same direction. Hence, after applying an even gamma the original direction can change.

Only after applying an even gamma the cosine of the constructed vectors is in the range [0,1], since all values will be positive.

To be honest, I do not understand how this metric is producing any good results. I guess I might just use any of the other metrics.

Thanks for your help, have a nice weekend

[MichaelRoeder]

Yes, you are right. I am not sure how I came to this conclusion :confused: I guess it is time for me to start my weekend :wink: Thanks for clarifying this point :+1:

Nevertheless, I would like to point out an additional argument: it is very unlikely that you have vectors that have a lot of negative (N)PMI values. The topics that are typically checked are not simply randomly generated but the result of a topic modeling algorithm. So most of them should have some relation to each other. If they have no relation, they mostly have a value around 0. In this area, it does not matter that much whether the value is positive or negative. The interpretation of a highly negative (N)PMI value would be: one word reduces the chances of the other word to be mentioned. Especially high negative values are very uncommon. Such a vector typically just means that the word has a very rare occurrence and (N)PMI values are known to perform not very well with artifacts like that.

However, you don't have to use C_A if you don't find it trustworthy :wink: