Open phoenix-asv opened 4 years ago
It seems that the problem is caused by presence of multiple modalities. artm.LDA
doesn't support the multimodal case very well (I'm not sure which modality it uses, if any). As a result, the scores are confused about which modality they belong to.
Quick and dirty workaround. Add the following before calling fit_offline
:
POSSIBLE_MODALITIES = ["@russian", "@english"]
for modality_name in POSSIBLE_MODALITIES:
lda._internal_model.scores.add(SparsityPhiScore(name=lda._sp_phi_score_name + modality_name, class_id=modality_name))
use it like so:
for modality_name in POSSIBLE_MODALITIES:
value = lda._internal_model.score_tracker[lda._sp_phi_score_name + modality_name].last_value
print(f"sparsity_phi for {modality_name}: {value}")
Additionally, you need to replace get_top_tokens
function.
def get_top_tokens(model, num_tokens=10, with_weights=False, modality_name=None):
"""
:Description: returns most probable tokens for each topic
:param int num_tokens: number of top tokens to be returned
:param bool with_weights: return only tokens, or tuples (token, its p_wt)
:return:
* list of lists of str, each internal list corresponds one topic in\
natural order, if with_weights == False, or list, or list of lists\
of tules, each tuple is (str, float)
"""
model._internal_model.scores.add(
TopTokensScore(name=model._tt_score_name, num_tokens=num_tokens, class_id=modality_name), overwrite=True)
result = model._internal_model.get_score(model._tt_score_name)
tokens = []
global_token_index = 0
for topic_index in range(model.num_topics):
if not with_weights:
tokens.append(result.token[global_token_index: (global_token_index + num_tokens)])
else:
result_token = result.token[global_token_index: (global_token_index + num_tokens)]
result_weight = result.weight[global_token_index: (global_token_index + num_tokens)]
tokens.append(list(zip(result_token, result_weight)))
global_token_index += num_tokens
return tokens
for modality_name in POSSIBLE_MODALITIES:
top_tokens = get_top_tokens(lda, num_tokens=10, modality_name=modality_name)
for i, token_list in enumerate(top_tokens):
print(f"Topic #{i}: {token_list}")
I cannot access wiki-enru dataset
to check right now. Does it help?
Thanks for fast reply! Workaround helped)
Although, the same problem appears on my own dataset:
1 language, vw file contains lines docID word1:cnt1 word2:cnt2 ... wordN:cntN
.
So it seams this problem is not caused or caused not only by presence of multiple modalities.
I am trying to follow tutorial at http://docs.bigartm.org/en/stable/tutorials/python_tutorial.html and run into some issues. Here is the sample code for training LDA model on wiki-enru dataset:
Then I am trying to get some scores and top words in each topic with this code:
and I get:
Transformation seams to be working fine, but I am not sure of it's quality. No warnings or errors messages shown in jupyter notebook.
Host: Linux
Version: 14c93c2086c7906ad14a4df2f2d58542b70e6686 (current stable)
Log file:
bigartm.WARNING