Evaluation on the new pipeline with RND, LDA, and BTM as the aspect models

[farinamhz]

One model is saving for each aspect model regardless of folds, so we have conflict in the evaluation of different folds. In fact, we should save n (nfold) models. Also, in the new pipeline, all the results of pytrec are coming into one data frame, and it is not becoming empty after each model. So they are affecting each other. These problems will be resolved immediately.

[farinamhz]

As shown in the figure below, BTM has the best evaluation results, LDA is good, and, as expected before, the RND model is the worst. Based on oov, we can not get 1.00 as a result for metric@n for n > size(dictionary). Also, the dictionary for LDA and RND is the same, but the dictionary of BTM is different and has more words, so metric@n, which n is 100 that in this case, is more than size(dictionary), may have different results than LDA and RND in metric@n.

[farinamhz]

@hosseinfani Could you please see the above results? I just want to make sure that they are reasonable now.

[hosseinfani]

@farinamhz thanks. make sense now. Just two notes: 1) Why btm's dict is different? 2) Make the three columns of figures into one, meaning we will have three bar with different colors for p@k, s@k, ...

[hosseinfani]

@farinamhz please update in (1) and (2)

[farinamhz]

@hosseinfani I updated the dictionary for BTM, and now we have the same dictionary from gensim for all three models, including LDA, BTM, and RND. However, the metrics for BTM decreased significantly on the new dictionary (gensim dictionary instead of BTM dictionary!

I will update part 2 soon to show my result in figures.

[farinamhz]

Hello @hosseinfani, Here are some examples of the new plots on semeval text version of 2016 for 25 topics. Also, we have a visualization code now in the pipeline that will create the plots for all the datasets and different values for the 'naspects' parameter in a directory named 'plots' under the 'output' directory.

• Precision_@K:

  

• Recall_@K:

  

• Mapcut@K:

  

• Ndcgcut@K:

  

• Success_@K:

  

As shown, BTM is not working correctly, so I will check and update you soon.

[hosseinfani]

@farinamhz redraw the figures in a hist x: #aspect y: k from 1:1:100 z: metric value

[farinamhz]

@hosseinfani By x, y, and z, do you mean a 3D histogram?

[hosseinfani]

@hosseinfani

By x, y, and z, do you mean a 3D histogram?

yep.

[farinamhz]

@hosseinfani Ok, I will update it tomorrow.

[farinamhz]

Hi @hosseinfani, This is the update of figures for the following parameters in XML dataset (Semeval 2016):

aspect, k from 5 to 50 with step = 5, and z: metric value

The pipeline is running on k from 1 to 100 with step=1 and aspects from 1 to 50 with step = 1, but it takes time to give the results.

• Precision_@K:

  • LDA:

  • BTM:

  • RND:

• Recall_@K:

  • LDA:

  • BTM:

  • RND:

• Mapcut@K:

  • LDA:

  • BTM:

  • RND:

• Ndcgcut@K:

  • LDA:

  • BTM:

  • RND:

• Success_@K:

  • LDA:

  • BTM:

  • RND:

[hosseinfani]

@farinamhz thanks. please fix the x and y in increasing order. Also, the k should be 1:1:100, right?

[farinamhz]

@hosseinfani I fixed them, and here is an example of the updated figures. However, I am waiting for the results on x: #aspect 1:1:50 y: k from 1:1:100 z: metric value

It seems that takes time.

• Example:

[farinamhz]

@hosseinfani Update with more aspects and K values:

• LDA model on XML dataset and Success values:

[hosseinfani]

@farinamhz I think we can close this issue. let me know otherwise.