Use language experience by LoC to cluster developers and projects in the same space

warenlg commented 5 years ago

Leverage the language distribution by LoC of both the developers and the repos of a given codebase, and cluster them showing similarities between one another. What has been done for now:

[x] Fetched all the src-d repos and extract statistics from them - done with hercules
[x] Reconstructed the identities using connected components
[x] Aggregated the language distributions of both the developers and the repos
[x] Filtered the small repos and contributors with commit thresholds
[x] Filtered auto-generated languages: JSON, SVG, CSV
[x] Normalizeed the distributions for each repos/devs
[x] Dimensionality reduction with umap and the euclidean metric
[x] Clustering first with KMeans then switch to DBSCAN
[x] Visualization with plotly online
[x] Added and then removed the bblfsh repos to the analysis (it worsened the results because of the many different clients and drivers)

warenlg commented 5 years ago

Results so far https://plot.ly/~warensourced/35/devproject-similarity-based-on-language-experience-loc-normalized-on-sourced-cod/#/

I'm now working on adding the cluster colors and the edges between the points to represent the contributions

vmarkovtsev commented 5 years ago

This is done.

src-d / ml-backlog

Use language experience by LoC to cluster developers and projects in the same space #77