Data Aggregation - Githubissues

WGierke / git_better

3rd-placed solution for the informatiCup2017

https://git-better.herokuapp.com/

Apache License 2.0

5 stars 3 forks source link

Data Aggregation #3

Closed WGierke closed 7 years ago

WGierke commented 8 years ago

Collect as many labeled repositories from as many sources as possible

Ideas:

GitHub Search API: see crawl.py for an example
GitHub.com/explore: e.g. repos in https://github.com/showcases/programming-languages can be classified as DEV with pretty high certainty

Google Search API : clone the repo and fire a query with e.g.

from google import google
results = google.search("site:.github.io", 1)

WGierke commented 8 years ago

Sources

Let's comment here where we got our processed data from to keep track of the sources:

30: original training data
9 DATA: manual Google search for open data repositories
82 DATA: repositories of Github user 'datasets'
17 EDU: Github Search for 'course, material'
17 DOCS: Github Search for 'documentation'
423 WEB: Google Search for 'site:.github.io'
58 HW: GitHub Search for "homework, assignments, solution"
13 DEV: Showcases Virtual Reality
12 DEV: Showcases Software Development Tools
14 DEV: Showcases Front-end JavaScript frameworks
20 DEV: Showcases DevOps tools
16 DEV: Showcases Text editors
24 DEV: Showcases Game Engines
27 DEV: Showcases Web Application Frameworks
42 DEV: Showcases Programming Languages
180 DOCS: GitHub Repo Content: Awesome Repos
6 DATA: Showcases Open Data
86 HW: Github Search for 'homework, solution'
--> 1242 processed repos

Current Distribution:

download 4