search

eXascaleInfolab / JUST

45 stars 7 forks source link

Can you share your evaluation code? #2

Closed rayrayraykk closed 4 years ago

rayrayraykk commented 4 years ago

Hello，thank you for your work first. When I tried to repeat your experimental results, I got the following results：

m 1 2 3    
macro-F1   0.868      
micro-F1   0.869      
NMI   0.32     DBLP
m 1 2 3    
macro-F1   0.5078      
micro-F1   0.556      
NMI   0.288     Foursquare
m 1 2 3    
macro-F1   0.4119      
micro-F1   0.4753      
NMI         Movie

It seems that the F1 score is basically consistent with your paper, but the NMI score is far below the score on your paper.The code folows:

#K-means
from sklearn.cluster import KMeans
from sklearn.metrics import normalized_mutual_info_score

kmeans = KMeans(n_clusters=10, max_iter=300, n_init=10, init='k-means++',
                    precompute_distances=True, n_jobs=-1)
kmeans.fit(datamat_x)
y_pr = kmeans.labels_
NMI = normalized_mutual_info_score(datamat_y, y_pr)
print(NMI)

In addition, why is the F1 score of dataset DBLP significantly higher than that of dataset Foursquare, but in the unsupervised task using K-means , the NMI score of dataset DBLP is higher than the NMI score of dataset Foursquare?Besides，DBLP 4 labels，Foursquare 10 labels，supposedly, the result should be the opposite.

So can you share your evaluation code？ Thank you again！

rayrayraykk commented 4 years ago

Here is my code https://github.com/rayrayraykk/JuSt_walks

As a large number of communitiesoften results in very small NMI values (which makes it hard for com-paring different methods), we thus select only the top two largestcommunities and the corresponding nodes in individual graphs inthis experiment.