FC-Louvain clusering fails for the provided example

[luav]

$ python3 fast_consensus.py -f examples/karate_club.txt --alg louvain

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "fast_consensus.py", line 97, in louvain_community_detection
    return cm.partition_at_level(cm.generate_dendrogram(networkx_graph, randomize=True, weight='weight'), 0)
AttributeError: module 'networkx.algorithms.community' has no attribute 'partition_at_level'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "fast_consensus.py", line 352, in <module>
    output = fast_consensus(G, algorithm=args.alg, n_p=args.parts, thresh=args.tau, delta=args.delta, procs=args.procs)
  File "fast_consensus.py", line 124, in fast_consensus
    communities_all = pool.map(louvain_community_detection, get_yielded_graph(graph, n_p))
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
    raise self._value
AttributeError: module 'networkx.algorithms.community' has no attribute 'partition_at_level'

[adityat]

Do you have python-louvain installed?

[luav]

You are right, this issue occurred when I used the community from networkx.algorithms instead of python-louvain, it might be helpful to comment in the imports like:

import community as cm   # python-louvain community

By the way, I realized that issue afterwards, a bit extended your algorithm (mostly I/O formats) and added it to the Clubmark benchmarking framework: https://github.com/eXascaleInfolab/clubmark/blob/master/algorithms/fast_consensus.py

Moreover, we submitted a paper to BigData '19, where your algorithm is evaluated together with almost a dozen others. Overall, it has high accuracy but is much less scalable than Louvain being not always robust to small network pertubations. There main (technical) issue is that convergence of the Fast Consensus Louvain (FCoL) significantly and non-linearly depends on the number of worker processes in the multiprocessing pool even fixing all the algorithm-related parameters (n_p, tau, delta), which should not happen in theory.
Namely, FCoL converges on amazon and dblp (https://snap.stanford.edu/data/#communities) within couple of hours for the default values of tau, delta and n_workers = n_p = 5, but it can not converge within several days (basically, it hangs) setting n_workers <= 2 for the same n_p = 5. In practice, the difference there is only in the time when the workers are started. It seems, when all workers are started simultaneously then Louvain yields (approximately) the same results and does not require any consensus-related processing at all (probably, they have a tie breaking strategy depending on a low-resolution timer, system time). But when the workers are started in distinct time and Louvain yields distinct partitions then your algorithm fails to converge (hangs) on the outlined datasets for the default tau, independently on delta (I tried all values in the range 0.02 .. 0.1 with the step 0.01) and setting n_p = 5. It seems, for n_p = 10 on the single worker the convergence also never happens on that datasets. So, you might need to investigate the outlined issue further and reconsider either the default tau or something else to ensure the proper convergence.

[adityat]

Thanks a lot for your feedback. I will look into the issues you mentioned for real world datasets and see if it can be improved. We are additionally doing an extensive analysis of the procedure on real world datasets so we hope to have this fixed soon.